Evidence digests

Verify a digest

1. Request the raw file. A raw-payload reference reads raw-snapshots@<commit>:<path>#source=<source>: an exact commit on the site’s private raw-snapshots branch, the gzipped bundle at that path, and the payload’s source. Raw payloads are archived privately and available on request: email data@aicomputetracker.com with the reference.
2. Decompress the bundle and find the entry in its evidence list whose source is the reference’s decoded #source= value.
3. Hash its rawTexts: a single text as its UTF-8 bytes; several in order, each as its UTF-8 byte length in decimal, a colon, then its bytes. The hex result is the digest the manifest lists.

# python3 verify_digest.py <bundle.json.gz> <the reference's #source= value>
import gzip, hashlib, json, sys, urllib.parse

bundle = json.loads(gzip.open(sys.argv[1]).read())
source = urllib.parse.unquote(sys.argv[2])
for item in bundle["evidence"]:
    if item["source"] != source:
        continue
    texts = item["rawTexts"]
    digest = hashlib.sha256()
    if len(texts) == 1:
        digest.update(texts[0].encode("utf-8"))
    else:
        for text in texts:
            body = text.encode("utf-8")
            digest.update(f"{len(body)}:".encode("ascii") + body)
    print(item.get("gpu", "-"), digest.hexdigest())

Recompute a day’s root

A day’s root is the RFC 6962 Merkle Tree Hash of its digests. The day's distinct digests, each as its 32 raw bytes, in ascending order of their lowercase hex; each leaf hash is SHA-256(0x00 || digest). Each interior node is SHA-256(0x01 || left || right). A list of one leaf hashes to its leaf hash; a list of n > 1 leaves splits after the first k, where k is the largest power of two smaller than n, and the root is the node over the two halves' roots.

# python3 verify_root.py digests.json <YYYY-MM-DD>
import hashlib, json, sys

def tree_hash(leaves):
    if len(leaves) == 1:
        return hashlib.sha256(b"\x00" + leaves[0]).digest()
    split = 1
    while split * 2 < len(leaves):
        split *= 2
    return hashlib.sha256(
        b"\x01" + tree_hash(leaves[:split]) + tree_hash(leaves[split:])
    ).digest()

day = next(d for d in json.load(open(sys.argv[1]))["days"] if d["date"] == sys.argv[2])
leaves = sorted({entry["sha256"] for entry in day["digests"]})
print(tree_hash([bytes.fromhex(d) for d in leaves]).hex() == day["root"])