Deterministic by Construction: How Attestrum Produces Byte-Identical Seals

Engineering documentation. Describes the concrete techniques that make Attestrum's corpus seal reproducible — given the same input bytes, the build emits a byte-identical manifest.parquet and an identical Merkle root across re-runs on the same machine (demonstrated over two full-corpus runs; see §4). Intended for partners, security auditors, and external verifiers evaluating Attestrum's reproducibility claims, and for engineers who want to apply the same disciplines elsewhere.

This is the "how" companion to cross-target-determinism.md, which catalogs which emitted fields are byte-identical across platforms (Linux glibc/musl, macOS, ARM) and documents the one field that is not. This document covers the mechanisms — the design rules that produce byte-identity in the first place — and reports a full-corpus, same-machine reproducibility experiment on WikiText-103.

Status: applies to the attestrum-pipeline seal path as of the Lookback Phase A commits. The §4 full-corpus double-seal experiment was run on 2026-05-31; the canonical seal was then signed, published to a public Hugging Face dataset, and re-verified with stock cosign (no Attestrum binary) on 2026-06-01 (see §4.2).

TL;DR

• A cryptographic provenance seal is only useful if it reproduces: a third party who re-runs the build on the same corpus must get the same Merkle root and the same signed bytes, or the signature proves nothing portable.
• Attestrum achieves this by construction. Seven disciplines — deterministic input ordering, pure transformations, parallel-but-order-stable accumulation, no wall-clock, no floating-point in the hash path, canonical serialization, and content-addressed storage — address the common sources of build-to-build drift.
• The claim is precise: given identical input bytes and these disciplines, the manifest.parquet and Merkle root are byte-identical across re-runs on the same machine. It is not a claim that any two unrelated inputs collide, nor that every downstream perceptual field is byte-stable (see the cross-target companion for the one exception), nor — on its own — a cross-platform claim (that is §3's matrix).
• Validated three ways: a fixture double-seal unit test, the 4-target cross-platform CI matrix, and — reported here — a full double-seal of the entire WikiText-103 corpus, sealed twice and compared byte-for-byte. (That double-seal ran on the corpus as first sealed — 824,850 passages; the detokenizer was then corrected and the corpus re-sealed to the current canonical 822,559 passages / root de95bddc…, see §4.1.)
• The loop is now closed on a live public artifact. The canonical seal is signed under Attestrum's keyless GitHub Actions identity and published at Attestrum/wikitext-103-sealed; its bundle was re-verified with stock cosign and curl alone — no Attestrum install — straight from the public Hub (§4.2).

1. Why byte-identity is the whole game

Attestrum compiles a training corpus into a signed bundle whose load-bearing claim is a single 32-byte Merkle root over the corpus. A verifier trusts that root only if they can reproduce it:

1. Take the same input corpus.
2. Re-run the seal.
3. Get the same root and the same manifest.parquet bytes.
4. Confirm the signature covers those exact bytes.

If step 2 produced even a one-byte difference — a map that iterated in a different order, a timestamp that leaked in, a float that rounded differently — the root would change and the verifier would (correctly) reject the bundle. Reproducibility is not a nice-to-have; it is the property that makes the signature mean anything.

sequenceDiagram
  autonumber
  participant P as Publisher
  participant S as Signed bundle
  participant V as Verifier
  P->>P: seal corpus → Merkle root R
  P->>S: sign R (Sigstore)
  S->>V: distribute bundle
  V->>V: re-seal the same corpus → root R'
  alt R' == R
    V->>V: signature covers reproducible bytes → ACCEPT
  else R' != R
    V->>V: cannot reproduce → REJECT
  end
  Note over P,V: the seal is only trustworthy if R is reproducible by anyone

So Attestrum treats any source of build-to-build non-determinism as a bug, and designs it out structurally rather than patching it after the fact.

2. The seven disciplines

flowchart TD
  IN["Input: Parquet shards<br/>(sorted filename order,<br/>rows in file order)"] --> SEG
  SEG["Pure segmentation + detokenization<br/>(deterministic function)"] --> HASH
  HASH["Parallel hash + CAS-put<br/>(rayon fold/reduce)<br/>each row stamps input_ordinal"] --> SORT
  SORT["Restore input order by input_ordinal,<br/>then canonical sort"] --> ROOT
  SORT --> MAN["manifest.parquet<br/>(sorted keys, zstd-3, pinned)"]
  ROOT["RFC 6962 Merkle root<br/>over sorted BLAKE3 leaves<br/>(no floats)"]
  MAN --> OUT["byte-identical output"]
  ROOT --> OUT

  classDef step fill:#1f3a5f,stroke:#4a90d9,color:#fff
  classDef out fill:#1f6f3f,stroke:#3ec072,color:#fff
  class IN,SEG,HASH,SORT,ROOT,MAN step
  class OUT out

2.1 Deterministic input ordering

The seal reads Parquet shards in sorted filename order and rows in file order. Directory-listing order is never trusted — the shard list is explicitly sorted before reading. The same set of files therefore always presents the same sequence of rows. (crates/attestrum-pipeline/examples/wikitext/seal.rs::sorted_shards.)

2.2 Pure, total transformations

Segmentation and moses-detokenization are pure functions: same string in, same passages out, no hidden state, no I/O, no clock. This is unit-tested directly (segment_is_deterministic). Because the transformation is pure, the only thing that can vary the leaves is the input — which §2.1 fixes.

2.3 Parallel work, order-stable output

Hashing ~10⁶ passages serially would be slow, so the hash + content-store-put stage runs in parallel (rayon par_iter().fold(...).reduce(...)). Parallelism is the classic source of non-determinism — work-stealing completes items in arbitrary order. Attestrum removes that risk with one move: each row stamps its original input index (input_ordinal) at construction time, before any parallel reordering. After the parallel stage the rows are re-sorted by input_ordinal to restore input order, then sorted again into the canonical on-disk order. The work-stealing schedule cannot leak into the output because the output order is recomputed from the stamped index, not from completion order. (crates/attestrum-pipeline/src/lib.rs::build_corpus.)

flowchart TD
  subgraph IN["Input order (fixed by §2.1)"]
    direction LR
    R0["row 0"]
    R1["row 1"]
    R2["row 2"]
    R3["row 3"]
  end
  R0 --> W
  R1 --> W
  R2 --> W
  R3 --> W
  W["rayon workers hash + CAS-put<br/>complete in ARBITRARY order<br/>each row carries its input_ordinal"]
  W --> ACC["per-worker Vecs merged<br/>(reduce — no shared lock)"]
  ACC --> RS["sort_by_key(input_ordinal)<br/>→ input order restored"]
  RS --> CS["canonical sort<br/>(document_id, occurrence_index)"]
  CS --> LV["leaves in fixed order"]
  LV --> RT["identical Merkle root<br/>(by design, independent of<br/>the worker schedule)"]

  classDef step fill:#1f3a5f,stroke:#4a90d9,color:#fff
  classDef out fill:#1f6f3f,stroke:#3ec072,color:#fff
  class R0,R1,R2,R3,W,ACC,RS,CS,LV step
  class RT out

This is why there is no Mutex<Vec> accumulator — a shared lock would serialize the workers and add contention without improving determinism; the fold/reduce + stamp + sort pattern is both faster and deterministic.

2.4 No wall-clock in sealed bytes

Nothing in the seal path reads the system clock. Where a timestamp is genuinely needed elsewhere in Attestrum, it is injected from a caller-supplied source_date_epoch (the Reproducible-Builds convention), never SystemTime::now(). The corpus manifest.parquet produced by this pipeline carries no clock-derived field at all (per-entry fetched_at is absent for the WikiText build), so there is no timestamp to diverge.

2.5 No floating-point in the hash or Merkle path

Every byte that feeds the Merkle root is computed with integer-only math: BLAKE3 and SHA-256 are integer hashes, and the RFC 6962 Merkle construction is pure byte concatenation + hashing with no arithmetic on reals. Floating-point is the one place where "the same operation" legitimately produces different bits on different toolchains/libms — so it is kept entirely out of the load-bearing path. (The single field that does use f32, and is therefore the one cross-platform exception, is documented in the cross-target companion — it is not part of the seal or the Merkle root.)

2.6 Canonical serialization

• The manifest is written by a Parquet writer pinned to a fixed configuration — zstd compression at a fixed level, no adaptive dictionary heuristics — so the same rows always compress to the same bytes. (crates/attestrum-manifest::writer_properties.)
• Any map serialized anywhere in the chain uses sorted keys / BTreeMap ordering, so key order is content-derived, never insertion- or hash-seed-derived.
• Leaves are extracted in the canonical sorted order (the same order written to the manifest), so the Merkle root is a function of the corpus viewed as a multiset, not of insertion or completion order.

2.7 Content-addressed storage

Each passage's bytes land in a content-addressed store keyed by its BLAKE3 digest. Content addressing is inherently order-independent: the same bytes always map to the same location, so the store's contents are a function of the corpus, not of the insertion order or the parallel schedule.

3. How we test it

The first two are continuous and small/synthetic. The third is the at-scale, real-data check — the experiment this document was written to record.

4. Experiment: full WikiText-103 double-seal

The double-seal below established the determinism property on the corpus as first sealed. The seal content was subsequently corrected (see §4.1), which changed the sealed bytes; the current canonical root is in §4.1. The result here stands as the same-machine byte-reproducibility proof, which is independent of the content fix.

Goal. Seal the entire WikiText-103 (raw) corpus twice, into two independent output directories on the same machine, and confirm the two manifest.parquet files and the two Merkle roots are byte-for-byte identical.

Input. wikitext-103-raw-v1, train split, 2 Parquet shards (~314 MB), pinned by SHA-256 (see ../lookback/corpus-source.md). After segmentation + the minimum-word floor, this pre-fix seal yielded 824,850 passages (one sealed leaf each). The detokenizer fix in §4.1 later dropped the ~2,291 mis-segmented stat-table glossary lines, giving the current canonical 822,559.

Method. Two runs of the seal generator (examples/seal-wikitext.rs, release build) over the identical input directory, into separate output dirs (run1, run2). Then: byte-compare the two manifest.parquet files (cmp), compare the two Merkle roots, and record wall-clock, peak RSS, and on-disk size for each run.

flowchart LR
  IN["WikiText-103<br/>2 Parquet shards<br/>(pinned SHA-256)"]
  IN --> S1["seal run 1<br/>→ run1/"]
  IN --> S2["seal run 2<br/>→ run2/"]
  S1 --> M1["manifest.parquet #1<br/>+ root R1"]
  S2 --> M2["manifest.parquet #2<br/>+ root R2"]
  M1 --> CMP{"cmp bytes<br/>and R1 == R2 ?"}
  M2 --> CMP
  CMP -->|identical| OK["reproducible at scale"]
  CMP -->|differ| BAD["determinism bug surfaced"]

  classDef step fill:#1f3a5f,stroke:#4a90d9,color:#fff
  classDef ok fill:#1f6f3f,stroke:#3ec072,color:#fff
  classDef bad fill:#6f1f1f,stroke:#d9534f,color:#fff
  class IN,S1,S2,M1,M2 step
  class OK ok
  class BAD bad

Results (2026-05-31).

• Merkle root identical across runs: ✅ YES (78cf503c…09d0e526)
• manifest.parquet byte-identical across runs: ✅ YES (cmp clean; identical SHA-256)
• Verdict: DETERMINISTIC_OK.

The decisive observation: the two runs took different wall-clock times (run2 was ~4% faster — less machine contention) yet produced byte-identical output — same root, same manifest down to the last byte. Execution timing and the parallel work-stealing schedule varied; the sealed bytes did not. That is determinism in the sense that matters for a signature: on this machine, the sealed output was determined by the input, not by how the machine got there. (The across-machine case is tested separately by the cross-platform matrix, §3.)

Content-addressed deduplication, observed. The corpus has 824,850 passages (leaves) but only 810,882 unique ones — ~13,968 passages are byte-for-byte duplicates of others (repeated boilerplate, identical sentences recurring across articles). Because the store is content-addressed (§2.7), duplicates collapse to a single object, while the manifest still records all 824,850 occurrences (each its own Merkle leaf, distinguished by an occurrence index). The Merkle root therefore covers the full corpus as a multiset, and both runs independently produced the same dedup count — itself another determinism data point.

Resource shape, confirmed: the build is fsync-bound — per run, ~1,090 s of system time (durable content-store writes) versus ~44 s of user CPU, with the ~94 min wall-clock dominated by waiting on those durable writes. Peak RSS (~2.5 GB) came in well under the ~5 GB pre-experiment estimate; the Parquet writer buffering its row group is the memory high-water mark.

4.1 Content-quality fix and re-seal (current canonical root)

The double-seal above proved the determinism property — that the pipeline emits byte-identical output for the same input. That property is independent of what text is sealed, so it carries forward unchanged through the content fix described here.

An adversarial review of the seal content (not its determinism) then found two defects in the WikiText-specific preparation, both since fixed:

• Detokenizer completeness. Measured over the full corpus, 39.65% of sealed passages carried a residual tokenization artifact a clean paste would not match — dominated by spaced straight quotes (33%), plus spaced slash, currency, and digit-colons. The detokenizer now handles these; the residual artifact rate fell to 0.10%.
• Backref attribution. WikiText-103 linearizes some stat-table glossaries as single-= lines mid-article, which were mis-read as article titles, mis-attributing the following passages and producing 328 colliding source_uri slugs. After rejecting those lines and disambiguating slugs, collisions are 0. (Leaf-level provenance was never affected — each leaf always carried its own source_url; this fixed the human-facing backref.)

Re-sealing the corrected corpus is the current canonical seal:

Because the pipeline is unchanged, this artifact is reproducible by the same disciplines.

Cross-platform reproduction (2026-05-31). The canonical root was first sealed on macOS/arm64. It has now been reproduced on Linux (a separate machine, OS, libc, and CPU architecture) — the property a Linux CI signer/verifier depends on. The lookback-seal-crosscheck workflow (workflow_dispatch, .github/workflows/lookback-seal-crosscheck.yml) runs the same seal-wikitext path on a GitHub Actions ubuntu-24.04 (x86_64/glibc) runner: it re-downloads the two pinned shards (SHA-256-verified against corpus-source.md), seals, and asserts the result against the canonical values. All three matched byte-for-byte:

The seal ran in 5 min 04 s wall-clock at ~2.3 GB peak RSS on the Linux runner (measured with /usr/bin/time -v, run #26726650991) versus ~94 min / ~2.07 GB on macOS: the durable-write (fsync) cost that dominated the macOS wall-clock (§4) is roughly an order of magnitude cheaper on the runner's Linux filesystem. Execution time differed ~19× — the sealed bytes did not differ at all. That is the determinism property a signature depends on, now confirmed across a change of OS and libc and CPU architecture in one run. This retired the seal-path-divergence gate that blocked signing: the artifact the CI keyless identity signs is produced by exactly the path proven here. Run: GitHub Actions lookback-seal-crosscheck #26725803592 (commit 372da9a). The corpus was then signed and published — see §4.2.

Practical takeaway. Because the bytes are platform-independent, where you seal is purely a speed choice, never an accuracy one. The recommendation for a production seal is therefore Linux — a cloud VM, CI runner, or Linux box — which is also where keyless signing runs, so seal and sign share one environment. macOS yields the identical root; it is slower only because it pays the full fsync durability cost on every content-store write.

4.2 Closing the loop: signed, published, verifiable with stock tooling

§1 framed the whole point: a seal is only trustworthy if a third party can reproduce its root and confirm the signature covers those exact bytes. As of 2026-06-01 the artifacts and the procedure for that are live and public — the §1 verifier path runs against public bytes with stock tools, which the project has exercised end-to-end (a genuinely outside audit is a separate, future step — see the close of this section).

What ran. A single workflow_dispatch pipeline (.github/workflows/lookback-publish.yml) sealed the corpus on a GitHub Actions ubuntu-24.04 runner — re-downloading the pinned shards, re-sealing via the seal-wikitext path — then signed the manifest keyless under Attestrum's GitHub Actions OIDC identity (a Fulcio certificate plus an immutable, public Rekor transparency-log entry) and published the artifact set to the public Hugging Face dataset Attestrum/wikitext-103-sealed.

Determinism is enforced as a gate, not just observed. Before the pipeline contacts Fulcio or Rekor, a pre-sign step asserts the freshly-sealed root, manifest.parquet SHA-256, and leaf count equal the canonical de95bddc… / eafa3dd7… / 822,559. A divergent seal aborts the run before any signature exists — §1's "REJECT" branch is wired into the publish path, so only the reproducible artifact can ever be signed.

Verification with stock tooling (no Attestrum install). Using only curl and stock cosign v3.0.6 — no Attestrum binary — the published bundle verifies straight from the public Hub:

base="https://huggingface.co/datasets/Attestrum/wikitext-103-sealed/resolve/main"
curl -sSL -o bundle.sigstore.json "$base/attestrum/bundle.sigstore.json"
curl -sSL -o manifest.parquet     "$base/attestrum/manifest.parquet"
cosign verify-blob-attestation \
  --new-bundle-format \
  --type 'https://attestrum.com/attestation/training-corpus/v0.3' \
  --bundle bundle.sigstore.json \
  --certificate-identity-regexp '^https://github\.com/Attestrum/Attestrum/\.github/workflows/lookback-publish\.yml@refs/.+$' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  manifest.parquet
# → Verified OK

The signature is bound to the Attestrum workflow identity (…/lookback-publish.yml@refs/heads/main, issuer token.actions.githubusercontent.com), which the command asserts — a different (e.g. personal) identity fails the same check. This is §1's verifier path, run with stock tooling. We ran it ourselves to confirm the published artifacts verify end-to-end; because it needs only curl + cosign, any party can repeat it without trusting Attestrum. To be precise about what this is and isn't: this is the project confirming its own pipeline against the live public bytes — not yet an outside audit. A genuinely third-party verification is the natural next external step.

One verification detail worth stating. cosign verify-blob-attestation defaults to the custom predicate type, and the cosign tested here (v3.0.6) rejects a non-default predicate type outright when --type is omitted. So the command names the predicate type (--type https://attestrum.com/attestation/training-corpus/v0.3). Without it the same bundle fails to verify on current cosign — a command-usability detail, not a signature problem. The CI acceptance gate passes the same --type.

Reproduce the root yourself. The determinism claim is checkable end-to-end without trusting this repository: re-fetch the two shards pinned by SHA-256 in corpus-source.md, run seal-wikitext over them, and you obtain de95bddc…; then confirm the bundle above covers that manifest. Root reproduced + bundle verified means the signature provably covers bytes you re-derived independently — which is the entire point of §1.

Run: GitHub Actions lookback-publish #26731092991; published commit 8978ccf.

5. What this proves — and its precise scope

Proves: for the same input bytes, on the same machine, the Attestrum seal pipeline produced a byte-identical manifest.parquet and an identical Merkle root at production scale, across the two runs — whose execution schedules differed (run2 was ~4% faster in wall-clock).

Scope and honest limits:

• This is same-input reproducibility — the property a verifier relies on. It is distinct from cross-platform byte-identity (different OS/arch/libc), which the cross-target companion covers separately and which has one documented exception (an f32-based perceptual hash, not in the seal path).
• It is not a collision claim about unrelated inputs; that rests on BLAKE3's collision resistance, a separate property.
• The experiment exercises one real corpus on one machine. The cross-platform matrix is what extends the byte-identity claim across toolchains and operating systems; the two results are complementary, not redundant.

6. References

Code

• crates/attestrum-pipeline/src/lib.rs::build_corpus — the fold/reduce + input_ordinal stamp + canonical sort that makes parallel hashing order-stable.
• crates/attestrum-pipeline/examples/wikitext/seal.rs — Parquet reader, passage→entry mapping, seal(), and the fixture double-seal determinism test.
• crates/attestrum-pipeline/examples/wikitext/segment.rs — pure segmentation + moses detokenization (segment_is_deterministic).
• crates/attestrum-merkle/src/lib.rs::merkle_root — RFC 6962 Merkle over sorted BLAKE3 leaves, integer-only.
• crates/attestrum-manifest/src/io.rs::writer_properties — the pinned Parquet writer configuration.

Diagrams

• docs/diagrams/lookback/wikitext-seal-pipeline.md — the seal generator pipeline (source-of-truth: code).
• docs/diagrams/sprint-3/rayon-pipeline.md — the parallel hash + CAS-put stage.
• docs/diagrams/lookback/wikitext-publish-pipeline.md — the gated build→sign→publish pipeline (§4.2).

Companion docs

• docs/research/cross-target-determinism.md — cross-platform byte-identity guarantees and the one documented exception.
• docs/lookback/corpus-source.md — the WikiText-103 source + per-shard SHA-256.

CI workflows

• .github/workflows/determinism.yml — the 4-target byte-identity matrix.
• .github/workflows/lookback-seal-crosscheck.yml — the Linux seal cross-check (§4.1).
• .github/workflows/lookback-publish.yml — the gated build→sign→publish pipeline (§4.2).

Live artifact

• Attestrum/wikitext-103-sealed — the signed, published corpus; verify with stock cosign per §4.2.