Corpus-Version Diffing: A Byte-Reproducible Delta Between Two Sealed Corpus States

Curation pipelines transform a corpus; data-version-control systems store its successive states. Neither cleanly answers a question every team eventually asks: what exactly changed between version N and version N+1 of my training mix — and can I prove it? attestrum diff is a read-only subcommand in the Attestrum CLI that answers precisely that, by comparing two already-sealed corpus manifests and emitting a deterministic delta report. Because both endpoints are cryptographically sealed, the delta is not a description you have to trust — it is evidence anyone can recompute.

1. What it is

attestrum diff <old> <new> is a deterministic, read-only diff between two already-sealed Attestrum corpus manifests. It has no write path, signs nothing, and mutates nothing — it reads two manifest.parquet files and reports what changed between them. A single run reports:

Documents added / removed / unchanged, by content identity.
Multiset (occurrence-count) shifts — a document that appeared 3× in the old corpus but 1× in the new one is unchanged in identity yet shifted in multiplicity.
Composition shift across five lenses — modality, source type, source dataset, SPDX license, and language — each as the old and new share of the corpus, over the union of labels in either version.
Per-version summaries — each version's BLAKE3 Merkle root, document count, distinct-document count, exact-duplicate count, and total bytes.

Output comes in two surfaces: a byte-reproducible report.json (full detail, canonical key ordering) and a concise human summary on stdout. Internally the comparison is a streaming merge-join over two canonically sorted manifest streams, so it never materializes either manifest in full — it scales to manifests with tens of millions of rows.

flowchart TD
    A["old manifest.parquet<br/>sealed · sorted by document_id"] --> V["validate<br/>schema version + path"]
    B["new manifest.parquet<br/>sealed · sorted by document_id"] --> V
    V --> J["streaming merge-join<br/>walk both in document_id order"]
    J --> C{"compare document_id"}
    C -->|in both| U["unchanged"]
    C -->|old only| R["removed"]
    C -->|new only| D["added"]
    U --> REP["deterministic report<br/>+ composition shift + both Merkle roots"]
    R --> REP
    D --> REP
    REP --> O1(["report.json"])
    REP --> O2(["stdout summary"])

2. Why it is valuable

The honest starting state for most corpus work is a hunch: "I think the dedup pass dropped some short documents," or "I'm fairly sure we rebalanced toward web text." attestrum diff turns that hunch into an auditable, byte-reproducible delta. Both endpoints are already sealed — each carries its own BLAKE3 Merkle root — so the delta between them is verifiable evidence, not a vibe. Anyone with the two manifests can recompute the identical report and get the identical bytes.

This fills a real gap. Data-curation frameworks transform a corpus — they filter, dedup, and rebalance, and the transformation is the product. Data-version-control systems like DVC and lakeFS store successive states with Git-like operations over files and objects. Attestrum replaces neither. But neither cleanly answers "what exactly changed between these two corpus states, in a report I can re-check byte-for-byte and stand behind as evidence?" File-level diffs tell you which blobs moved; they do not give you a content-addressed, composition-aware, independently reproducible delta tied to a cryptographic root on each side.

There is also a compliance dimension. The EU AI Act's Article 53(1)(d) obliges general-purpose AI model providers to publish a sufficiently detailed summary of training content. When a published corpus is revised, "here is exactly how the composition changed from the previous version" is a question such a summary may need to answer — and a per-source, per-license, per-language share shift that recomputes byte-for-byte is a clean way to back it.

One caveat, stated up front: attestrum diff measures a change. It does not judge whether the change improved the data. The interpretation is yours.

3. How to use it

The command takes two sealed manifests, old then new:

attestrum diff <old.parquet> <new.parquet> [--out <report.json>] [--timestamp <string>]

--out <path> writes the full deterministic report.json. Omit it for just the stdout summary.
--timestamp <string> embeds a Reproducible-Builds-style timestamp verbatim. Absent it, the report carries no wall-clock at all — which keeps the output byte-identical across runs and machines.

Exit codes mirror attestrum inspect:

Code	Meaning
`0`	success
`2`	argument error — a path is missing or not a file
`8`	schema-version mismatch — a manifest's schema version is unsupported
`1`	other read error

Worked example (genuine output)

Two versions of a small corpus. v1 = {doc01, doc02, doc03}. v2 is a curation pass that edited doc01, kept doc02, dropped doc03, and added doc04:

$ attestrum diff v1/manifest.parquet v2/manifest.parquet --out report.json
attestrum diff: wrote report.json
corpus diff (attestrum-diff-report/0.1)
identity mode: content-addressed multiset over BLAKE3 document_id; no "modified" category (a changed document is a removed old hash plus an added new hash)
old: fcdee42e76e055cb4090eec18919caef3475722dd31dc06b3c5e1d0c09a0dd92  (3 docs, 3 distinct, 645 bytes)
new: 60f9043815eea1a07dca6e53d77960a4e22a20daee5c98369e928b67ec0b94cd  (3 docs, 3 distinct, 406 bytes)
delta: added 2 · removed 2 · unchanged 1 · multiset-shifts 0

Read the counts carefully, because they teach the whole model. The delta is added 2 / removed 2 / unchanged 1 — not the "1 modified, 1 removed, 1 added" you might expect:

doc01 was edited. Its content changed, so its BLAKE3 content hash changed. The diff reports its old hash as one removed and its new hash as one added. There is no line connecting them — and that is deliberate.
doc03 was dropped → one more removed.
doc04 was added → one more added.
doc02 is byte-for-byte identical → the single unchanged.

The two distinct Merkle roots name the two corpus states cryptographically; the byte-count drop (645 → 406) reflects the smaller revised set. The identity model in one picture — an edited document is two events, not one:

flowchart LR
    E["doc01<br/>content edited"] --> O["its old content hash<br/>→ removed"]
    E --> N["its new content hash<br/>→ added"]
    O -. "no link between them — a changed<br/>content hash IS a different document" .-> N

4. When to use it

After a curation pass — dedup, filtering, or source rebalancing — to see precisely what came out and what the composition shift was.
When re-cutting a training mixture and you need the delta between the old mix and the new one.
In CI, to gate or document corpus changes: diff the previous sealed manifest against the new one, and fail the build or attach the report when the delta crosses a threshold you care about.
When auditing what a data pipeline actually did versus what it was supposed to do — the report is ground truth, independent of the pipeline's own logs.
When comparing two published versions of a dataset, to produce a reproducible, evidence-grade statement of how they differ.

5. Why we made it — design philosophy

The honesty model is the centerpiece. A manifest's document_id is the BLAKE3 content hash of the document. So an edited document is reported honestly as one removed (its old hash) plus one added (its new hash) — never a fuzzy-matched "modified" verdict. There is no caller-stable id in the manifest schema that could link an old version of a document to a new one, so the tool does not pretend there is. It states its identity mode in every report rather than guessing at a relationship it cannot prove.

The report declares its own limits, in-band. Every report carries an explicit list of what it deliberately does not do in this version:

No "modified" category — that would require a caller-stable document id, a change to the protected manifest schema.
No near-duplicate-rate delta — per-document fingerprints are not persisted in the manifest, so a fuzzy near-duplicate shift is out of scope for this subcommand.
No signed corpus-delta artifact yet — the diff is the reversible, unsigned leaf, deliberately not a frozen, signed predicate.

Determinism is the one absolute promise, and it is CI-enforced. Same two manifests → byte-identical report on any machine. The merge-join walks in document_id order, every accumulator is a BTreeMap, every example list is sorted by construction, shares round to six decimals for stable float formatting, and JSON rendering rides a shared canonical-key primitive. A committed golden plus a double-run byte-compare gate this in CI on every push.