attestrum compose is a read-only subcommand in the Attestrum CLI that summarizes what a sealed corpus is made of — its language, source-type, SPDX-license, and modality mix — and emits a deterministic composition report. What is actually inside this training corpus, in numbers anyone can re-check byte-for-byte? Because the manifest it reads is cryptographically sealed, the summary is not an assertion you have to trust — it is a computed fact tied to a corpus root.
1. What it is
attestrum compose --manifest <manifest.parquet> is a deterministic, read-only summary of a single sealed Attestrum corpus. It has no write path, signs nothing, and mutates nothing — it streams one manifest.parquet and reports what the corpus is composed of, over the documents actually included in training:
- Modality mix — text / image / audio / video / pdf / other.
- Source-type mix — crawl / public-dataset / private-licensed / user / synthetic / other.
- SPDX-license mix and language mix — the license identifier and BCP-47 language tag carried by each document.
- Two weights per bucket — a document count and a
size_bytessum, each as a percentage of the included totals. Byte-weighting answers "how much of the training data," count-weighting answers "how many documents." - A coverage % per dimension — source-type, license, and language are optional in the manifest, so each carries the share of documents that actually declare a value. A
Nonelands in an explicitunspecifiedbucket and is excluded from coverage — never silently dropped. - A corpus anchor — the BLAKE3 Merkle root over every document, byte-identical to the root the seal pipeline produced, plus included / excluded counts and total bytes.
Output comes in two surfaces: a byte-reproducible report.json (full detail, canonical key ordering) and a human-readable report.md with one table per dimension. The walk is a single streaming pass over the manifest's batch reader, so it never materializes the manifest in full and scales to tens of millions of rows.
flowchart TD
M["sealed manifest.parquet<br/>18-column, schema v2"] --> S["stream rows<br/>(constant-memory batch reader)"]
S --> L["collect every document_id<br/>→ recompute BLAKE3 Merkle root"]
S --> I{"included == true?"}
I -->|yes| H["histogram by modality / source-type /<br/>SPDX-license / language<br/>weighted by (count, bytes)"]
I -->|no| X["excluded count"]
H --> R["deterministic report<br/>+ per-dimension coverage %"]
L --> R
X --> R
R --> O1(["report.json"])
R --> O2(["report.md"])2. Why it is valuable
The compliance driver is concrete and near. The EU AI Act's Article 53(1)(d) obliges general-purpose AI model providers to publish a sufficiently detailed summary of training content, and the Commission has published a mandatory template for it; California's AB 2013 adds a state-law disclosure obligation effective 2026. Both ask, in effect, what is this corpus made of? — by source, by kind, by license. attestrum compose answers that from the sealed corpus itself, so the published summary is a computed fact tied to a cryptographic root, not a hand-written assertion a reader has to trust.
This fills a gap the existing tooling leaves open. Data-curation frameworks like Dolma and datatrove build a corpus — they filter, dedup, and mix sources, and statistics are a by-product of the pipeline run. But a pipeline's own logs are not independently checkable, and they describe the process, not a frozen, verifiable artifact. attestrum compose computes the composition of the sealed manifest after the fact, so the same manifest yields the same summary for anyone — the corpus author, an auditor, or a regulator — with no need to re-run or trust the original pipeline.
Two caveats, stated up front. First, attestrum compose measures composition; it does not judge it. A mix that leans heavily on web crawl or on synthetic text is reported neutrally — the interpretation is yours. Second, the summary is only as complete as the manifest's metadata: if a corpus was sealed without language or license labels, compose reports that honestly as low coverage rather than inventing values. The coverage % is the feature that keeps the gap visible.
3. How to use it
The command takes one sealed manifest:
attestrum compose --manifest <manifest.parquet> [--out <dir>]--manifest <path>— the sealedmanifest.parquetto summarize.--out <dir>— directory to writereport.json+report.mdinto (default: the current directory).
The report carries no wall-clock — it is a pure function of the manifest, which is what keeps the output byte-identical across runs and machines.
Worked example (genuine output)
Take a small five-document corpus: four documents are included in training (a mix of text and image, two crawled and one from a public dataset, with assorted licenses and languages) and one is recorded but excluded. Running compose over its sealed manifest:
$ attestrum compose --manifest corpus/manifest.parquet --out out
compose: 5 document(s) (4 included · 1 excluded) → 2 modality / 3 language bucket(s)
out/report.json
out/report.mdThe human-readable report.md (genuine, unedited):
# attestrum compose — training-content summary (v0.1.0)
- manifest: corpus/manifest.parquet
- corpus root (BLAKE3 Merkle): `3eb572819de9fac6c5216ce8989fa2bd3cfb8965188aa3281ac7ad175a0e3e19`
- documents: 5 (4 included · 1 excluded)
- included size: 750 bytes
## Modality (coverage 100.0% by count · 100.0% by bytes)
| bucket | docs | % docs | bytes | % bytes |
|---|---:|---:|---:|---:|
| text | 3 | 75.0% | 450 | 60.0% |
| image | 1 | 25.0% | 300 | 40.0% |
## Source type (coverage 75.0% by count · 80.0% by bytes)
| bucket | docs | % docs | bytes | % bytes |
|---|---:|---:|---:|---:|
| crawl | 2 | 50.0% | 300 | 40.0% |
| public_dataset | 1 | 25.0% | 300 | 40.0% |
| unspecified | 1 | 25.0% | 150 | 20.0% |
## License (SPDX) (coverage 50.0% by count · 53.3% by bytes)
| bucket | docs | % docs | bytes | % bytes |
|---|---:|---:|---:|---:|
| unspecified | 2 | 50.0% | 350 | 46.7% |
| CC-BY-4.0 | 1 | 25.0% | 100 | 13.3% |
| CC0-1.0 | 1 | 25.0% | 300 | 40.0% |
## Language (coverage 75.0% by count · 60.0% by bytes)
| bucket | docs | % docs | bytes | % bytes |
|---|---:|---:|---:|---:|
| en | 2 | 50.0% | 300 | 40.0% |
| fr | 1 | 25.0% | 150 | 20.0% |
| unspecified | 1 | 25.0% | 300 | 40.0% |Read the tables with the coverage line in mind. The corpus is 100% text-and-image by modality (modality is always present), but only 50% of documents declare a license — the unspecified row and the coverage 50.0% by count header say the same thing two ways, and neither hides it. The excluded document never appears in any composition bucket: the summary describes the training content, and the byte totals (750 included of a larger sealed set) make the inclusion boundary explicit. The Merkle root names this exact corpus state, so the whole summary is pinned to one verifiable artifact.
The honesty rule in one picture — an absent label is a bucket and a coverage hit, never a silent drop:
flowchart LR
D["a document row"] --> Q{"language declared?"}
Q -->|"en"| K["counts into the 'en' bucket<br/>AND raises coverage"]
Q -->|null| U["counts into the 'unspecified' bucket<br/>but NOT toward coverage"]4. When to use it
- When publishing an Article 53 / AB 2013 training-content summary — generate the composition from the sealed corpus so the disclosure is a computed, re-checkable fact.
- Before and after a curation pass — run compose on each version to see how the mix moved (pair it with
attestrum difffor the document-level delta). - When evaluating a third-party corpus — get an independent composition readout from the published manifest, without trusting the publisher's own statistics.
- In CI, to assert a corpus's composition stays within bounds you care about — e.g. fail the build if synthetic-text share crosses a threshold, or if license coverage drops.
5. Why we made it — design philosophy
Coverage is honesty made measurable. Most composition tools quietly drop rows with missing metadata, which makes a thinly-labeled corpus look cleaner than it is. attestrum compose does the opposite: a missing source, license, or language is bucketed as unspecified and excluded from the coverage figure, so every dimension reports the share of the corpus it can actually speak to. A 50%-license-coverage corpus is reported as exactly that — the gap is a number, not an omission.
Two weights, because the regulator's question is about proportion of data, not count of files. A corpus can be 90% of documents but 10% of bytes in one language. Reporting both count-share and byte-share keeps either framing from misleading: byte-weighting answers "how much of the training data," which is closer to what a training-content summary is really asking.
The summary is pinned to a cryptographic root. Compose recomputes the BLAKE3 Merkle root over every document in canonical order — byte-identical to the root the seal pipeline produced — and prints it on the report. So the composition is not a free-floating description; it is bound to one specific, verifiable corpus state that anyone can check.
The report declares its own limits, in-band. This version computes a composition summary; it deliberately stops short of two things, and says so: there is no signed composition predicate yet — compose is the reversible, unsigned leaf, not a frozen signed attestation — and it does not emit the Commission's exact Article 53 template yet, only the underlying composition the template is filled from. Both are deferred, schema-affecting work, kept out of the leaf on purpose.
Determinism is the one absolute promise, and it is CI-enforced. Same manifest → byte-identical report on any machine. Every histogram is a BTreeMap, percentages round to six decimals, the manifest is walked in canonical order, and JSON rendering rides the same shared canonical-key primitive as the rest of the CLI. No wall-clock, ever. A committed golden plus a double-run byte-compare gate this in CI on every push.
That is the whole intent: a small, single-purpose instrument that turns a sealed corpus into a defensible statement of what it contains, makes its own blind spots legible, and produces the same bytes every time.