ADR-003: Corpus Package Format

• Status: Conditional; accepted only when F2-003-W01 is accepted and landed on main
• Date: 2026-08-18
• Roadmap task: F2-003
• Decision owners: corpus engineering and data release maintainers

Context

The accepted 2026 Turkish corpus currently exists across source files and a verified legacy database. Rebuilding a new database by repeatedly matching those mutable surfaces would make import correctness depend on runtime heuristics. A database dump would preserve engine internals rather than a reviewable, portable corpus contract.

ADR-001 requires exact core reconstruction without annotations. ADR-002 requires deterministic bootstrap identities from a versioned package issuance contract. The canonical corpus package must therefore be readable across runtimes, streamable at roughly 1.7 million token occurrences, deterministic across clean builds and strict enough to detect one changed codepoint, row, ordering fact or parent relation.

This ADR fixes the release envelope and serialization. ADR-004 and ADR-005 will fix the token and span row semantics before F3-005 writes their JSON Schemas.

Decision

Release bundle and component boundary

The canonical publication unit is munderecat-corpus-bundle/v1. Its exploded form is authoritative for validation:

corpus-release-v1/
|-- manifest.json
|-- checksums.sha256
|-- core/
|   |-- works.jsonl
|   |-- text-versions.jsonl
|   |-- books.jsonl
|   |-- chapters.jsonl
|   |-- paragraphs.jsonl
|   |-- sentences.jsonl
|   `-- token-occurrences.jsonl.zst
`-- companions/
    |-- provenance-bindings.jsonl.zst
    `-- legacy-id-crosswalk.jsonl.zst

The core/ component contains only the ADR-001 core release envelope, hierarchy, exact content, boundaries and ordering needed for reconstruction. Headings, segments, notes, compounds, Ottoman renderings, lexical analysis, dictionary data, repetition, alignment and citations are forbidden even if a legacy database stored them next to core rows.

companions/provenance-bindings.jsonl.zst is required for a published release. It binds core identities and identity paths to accepted source manifests and compiler evidence, but core reconstruction cannot read it. legacy-id-crosswalk.jsonl.zst is optional in the general format and required for the initial legacy migration. It is a typed migration aid, not public identity or core content.

The initial release consumes an accepted munderecat-corpus-public-provenance-v1 handoff; the packager cannot synthesize its process, evidence, closure or public manifest identities. Coverage-set rows use eight compact bindings. Their membership_artifact_sha256 identifies a public ID-only JSONL projection, and their count/root values describe the per-kind ordered ID sets in that artifact. The richer stable-identity authority contains source locators and remains private; it cannot be substituted for the public membership artifact or disclosed by hash as an indirect private commitment.

The handoff also publishes an order-independent legacy mapping-set root over lexicographically sorted [crosswalk_id,row_sha256] pairs. This proves that package sorting did not add, remove or reinterpret a mapping. F3-012 source-evidence and chunk roots remain private and are represented publicly only by an opaque accepted attestation. Every public field, limitation and artifact fact is covered by the handoff’s public projection root; the private graph and private/public mapping are outside the release payload.

Handoff occurrence IDs are generated afresh for each materialization and retained from that successful run; they are not configuration constants or deterministic rebuild IDs. The verifier streams the accepted identity and crosswalk authorities again and compares the disclosed projections to those exact rows. A candidate is renamed to its no-replace final path only after this independent staged verification, closure/attestation validation, disclosure checks and file-mode checks succeed.

Each component has an independent root. The outer manifest binds all present components, so changing a companion changes the bundle identity while leaving the core release digest unchanged. Annotation releases use separate package contracts and cannot be added under core/ or disguised as companions.

The file names above are the v1 required core streams. Later row schemas may prove that another exact core stream is necessary, but adding or removing a core stream changes the package format major version. A namespaced companion extension may be added only when the manifest declares its schema, ownership and component root and core reconstruction remains independent from it.

JSON and JSON Lines

manifest.json is one RFC 8785 canonical JSON object followed by LF. Every .jsonl logical stream consists of one RFC 8785 canonical JSON object per line, each followed by exactly one LF. Files are UTF-8 without BOM; CRLF, blank lines, comments, NaN, infinity, duplicate object keys and unpaired Unicode surrogates are invalid.

Source text values are not Unicode-normalized. Their decoded Unicode scalar sequence must equal the accepted source sequence, including punctuation, whitespace, apostrophes, diacritics, Arabic passages and separator characters. The source-input manifest declares how original bytes are decoded; reconstruction digests use the exact UTF-8 encoding of the accepted scalar sequence, not the original file’s possibly non-UTF-8 transport bytes. JSON escaping is transport syntax only. A line break that belongs to source text is encoded inside a JSON string; it never becomes an extra JSONL record delimiter.

Numbers follow RFC 8785 interoperability constraints. Identity and position fields use non-negative integers no larger than 9007199254740991; a schema needing a larger exact integer uses the canonical unsigned decimal string grammar from ADR-002. Row schemas forbid undeclared fields.

JSONL is selected because it supports streaming validation/import, exact line-level diagnostics, cross-language implementations and useful review diffs for all small and medium streams. It is a release interchange format, not a claim that JSONL is the most efficient runtime query representation.

Row payload and row hash

Every row has a public id, a declared entity kind and row_sha256. Core child rows also carry parent identity and ordering facts; exact field names are owned by the later stream schemas.

row_sha256 is not self-referential. Its input is the RFC 8785 serialization of this canonical envelope, where payload is the complete row with row_sha256 omitted:

{"algorithm":"sha256-rfc8785-row-v1","entity_kind":"sentence_occurrence","payload":{}}

The lowercase hexadecimal SHA-256 of those bytes is stored as row_sha256, then the complete row is canonicalized for its JSONL line. Any field change therefore changes the row hash, while field order or insignificant JSON whitespace cannot create an alternative valid representation.

Each stream schema declares one deterministic traversal sort tuple. Core streams sort by ancestor positions followed by local zero-based position, with stable ID as a final corruption-detection tie breaker, never as the primary corpus order. The compiler rejects duplicate or non-contiguous sibling positions before writing. File system enumeration, database query order and worker completion order are not sort inputs.

Logical and transport bytes

Small core streams remain uncompressed .jsonl. The fixed large streams token-occurrences, provenance-bindings and legacy-id-crosswalk use one .jsonl.zst file each. Compression is never selected dynamically by current size; that would change format when the corpus grows.

For every stream, the manifest records:

• component and relative POSIX path;
• content role and entity kind;
• immutable schema ID and schema SHA-256;
• media type and compression profile;
• row count and logical byte count;
• SHA-256 of the exact logical JSONL bytes;
• SHA-256 and byte count of the stored transport bytes; and
• an ordered aggregate of row IDs and row hashes.

For uncompressed files, logical and transport bytes and hashes are equal. For zstd, the logical hash is computed over the exact decompressed JSONL stream and is the content authority; the transport hash identifies the released compressed artifact. An importer verifies both.

The zstd-frame-v1 profile is one frame, compression level 9, one worker, content size and frame checksum enabled, no dictionary and no optional filename, timestamp or host metadata. The release toolchain pins the compressor version and executable digest. Recompression with another implementation may preserve logical content but creates a different transport artifact and cannot masquerade as the released bundle.

Aggregate roots without cycles

Roots use canonical JSON envelopes rather than ambiguous byte concatenation.

For each stream, row_root_sha256 hashes:

{"algorithm":"sha256-ordered-rows-v1","rows":[["<id>","<row_sha256>"],["<id>","<row_sha256>"]]}

in exact stream order. Zero-row streams use an empty rows array. A component root hashes this canonical envelope, where files is sorted by relative-path UTF-8 bytes:

{"algorithm":"sha256-component-v1","component":"core","files":[["<relative_path>","<schema_id>","<schema_sha256>",0,"<logical_sha256>",0,"<row_root_sha256>"]]}

Each file tuple is [relative_path, schema_id, schema_sha256, logical_byte_count, logical_sha256, row_count, row_root_sha256]. Schema identity is therefore part of content identity: identical JSON bytes interpreted by a different schema cannot retain the same component root. The core_root_sha256 is the core component root and is the immutable content digest for ADR-001. Companion roots use the same algorithm with their declared component name.

The outer bundle_root_sha256 hashes this canonical envelope, with components sorted by component-name UTF-8 bytes:

{"algorithm":"sha256-bundle-v1","components":[["core","<component_root_sha256>"],["provenance","<component_root_sha256>"]]}

It contains the core root and every declared companion root and excludes manifest.json and checksums.sha256, avoiding a cycle. manifest.json stores all inputs and roots but does not store a hash of itself.

checksums.sha256 contains lowercase transport SHA-256, two ASCII spaces and a safe relative POSIX path for manifest.json and every component file, sorted by UTF-8 path bytes and terminated by LF. Paths are unique and cannot contain control characters, backslashes, empty segments, . or ... The file excludes itself. The release registry records the SHA-256 of exact manifest.json, exact checksums.sha256 and the final archive, so all envelope bytes remain externally addressable without self-reference.

Manifest contract

manifest.json contains at least:

• format ID and major version;
• package ID, core release ID and ADR-002 identity_release_key;
• identity schema ID, namespace and identity-path manifest hash;
• source-input manifest IDs and hashes;
• compiler source commit, locked environment identity and compiler artifact hash;
• encoding and documented source-normalization policy;
• compression and archive profile IDs plus reference-tool artifact hashes;
• schema ID and hash for every stream;
• complete stream inventory and expected row counts;
• component roots and bundle root;
• reconstruction profile ID and hash plus expected sentence, paragraph, chapter and release reconstruction digests; and
• required importer capability version.

It contains no wall-clock build time, host path, username, UID/GID, container ID, credential, random build identifier or database sequence. Operational timestamps belong to external release evidence. Semantically identical clean builds produce byte-identical manifests.

Schema references are content-addressed and must resolve to version-controlled files in the release source. Unknown package major versions, unknown required capabilities, unknown core streams or schema hash disagreement fail before row import.

The v1 digest profile is sha256-reconstruction-utf8-v1. Each reconstructed entity records its exact UTF-8 byte count and lowercase SHA-256 of those bytes. Each sentence, paragraph, chapter and release aggregate hashes an RFC 8785 canonical envelope containing the algorithm ID, entity kind and ordered [entity_id, utf8_byte_count, text_sha256] tuples. The content-addressed reconstruction profile defines how row fields and separators assemble those exact bytes; its ID and hash must be fixed before a package can validate. ADR-004 may define that assembly algorithm but cannot change this framing without a new profile.

Deterministic archive transport

The publishable transport is corpus-release-v1.tar.zst, containing exactly one top-level corpus-release-v1/ directory and the exploded bytes above. The inner tar uses the byte-level tar-ustar-v1 profile. Members are the required directory entries and regular files in relative-path UTF-8 byte order; directory names end in / and precede their descendants. Files use typeflag 0 and mode 0644; directories use typeflag 5, mode 0755 and size zero. UID, GID and mtime are zero. The magic bytes are ustar plus NUL and version is ASCII 00; link name, owner/group names, device fields and header padding are all NUL. Name and prefix fields are NUL-padded. Mode, UID and GID are seven zero-padded ASCII octal digits plus NUL; size and mtime are eleven such digits plus NUL; base-256 is forbidden. The checksum field is eight spaces while summing unsigned header bytes and is emitted as six zero-padded octal digits, NUL and space. Path name/prefix splitting uses the longest slash boundary that fits both fields. File payloads are padded with NUL to 512-byte blocks and exactly two all-zero blocks terminate the archive, with no trailing blocks. Extended attributes, ACLs, sparse files, devices, hardlinks, symlinks and PAX/GNU extensions are forbidden.

The outer zstd stream uses zstd-frame-v1. All paths must fit the selected ustar limits; path shortening is a format design action, not an automatic PAX fallback. The release toolchain pins the tar-ustar-v1 reference encoder source, artifact hash and exact invocation in the same environment lock as zstd. The exploded form and archive must contain the same declared file bytes. Two clean builds compare core root, bundle root, manifest bytes, checksum bytes and final archive SHA-256.

Safe and atomic import

Candidate validation and authoritative publication are distinct operations. A candidate may be checked against a caller-supplied expected hash but gains no public status. Publication starts only from an immutable accepted entry in the importer’s configured trusted release registry; that entry binds package ID, core release ID, identity release key, manifest hash and archive hash. Package-provided or caller-provided registry claims cannot authorize publication.

An authoritative importer performs these stages in order:

1. Resolve the accepted registry entry and verify its exact archive hash before extraction.
2. Apply importer-owned ceilings to archive bytes, expanded tar bytes, member count, per-member bytes and total extracted bytes while streaming outer zstd and tar into a private staging directory. Reject excess before allocation or write, plus absolute paths, .., backslashes, duplicate/case-colliding paths, links, devices and undeclared entries.
3. Validate exact file set, manifest canonical bytes, checksums.sha256 and every identity/hash bound by the accepted registry entry.
4. Apply importer-owned ceilings and the stricter declared limits to each inner compressed size, logical size and row count while streaming zstd; excess or truncation fails before database publication.
5. Validate every line against its exact schema, canonical rendering, row hash, stream order and aggregate roots.
6. Validate UUID versions and identity paths, unique IDs, parent containment, contiguous positions, cross-file references and expected reconstruction digests.
7. Load only into isolated staging tables and compute the resulting core digest.
8. In one transaction, acquire uniqueness-backed claims for package ID, core release ID and identity release key, compare their bound roots/contracts, attach all validated companions and bundle metadata, then make the complete outer package visible. Any disagreement or concurrent conflict rolls back every claim and removes staging state; readers never observe a core-only partial package.

Import is idempotent at two distinct identities. Reusing a package ID requires the identical manifest and bundle root. Reusing a core release ID requires the identical core root, complete core schema set and identity contract but permits a different package ID whose companion set differs. Reusing an identity_release_key with different source-input, identity-schema or entity-path manifests remains a hard conflict under ADR-002. These checks use database uniqueness constraints and compare-and-publish semantics in the publication transaction, not an earlier best-effort lookup. A companion failure prevents publishing the outer release bundle, but no companion can repair or alter staged core rows.

The importer never reads raw legacy files, calls a fuzzy matcher or consults an annotation service. Those operations belong to the one-time compiler and separate annotation pipelines.

Repository and release retention

Schemas, small fixtures, manifests and intentionally reviewable small release components may live in Git. Full corpus archives are content-addressed release artifacts unless a later size audit explicitly approves Git storage. Raw source trees, database dumps and SQLite files are never embedded in the bundle.

The release registry retains archive locator, size and SHA-256 plus manifest, checksum, bundle and core roots. A lost transport can be rebuilt byte-for-byte from the accepted source, compiler and lock; a mirror with matching logical content but different transport bytes is not the same released artifact.

Consequences

Benefits

• Core can be reviewed and imported without PostgreSQL dump compatibility or legacy runtime matching.
• Exact codepoints and ordering have row, stream, component and release-level integrity checks.
• Large streams remain practical while their decompressed authority is independently verifiable.
• Provenance and legacy crosswalks remain bundled and traceable without contaminating the core digest.
• ADR-004/ADR-005 can evolve row schemas within an already fixed deterministic envelope and release process.

Costs

• The release toolchain must implement canonical JSON, pinned zstd and normalized ustar rather than relying on default serializers and archive commands.
• Manifest, checksum and multiple aggregate roots deliberately duplicate some integrity metadata for streaming diagnosis and independent verification.
• JSONL is larger and less query-efficient than columnar formats before compression.
• Any package-major or identity-path change requires explicit migration and cannot be hidden as a compiler refactor.

Alternatives Considered

PostgreSQL dump as canonical package

Dumps bind the release to engine/version details, object ownership and import order and are difficult to inspect outside PostgreSQL. They remain backup artifacts, not the corpus interchange authority.

Parquet for all streams

Parquet compresses and scans well but introduces a larger cross-runtime type, ordering and metadata surface for the first canonical contract. Runtime analytics may derive Parquet from the package; it is not the v1 authority.

One uncompressed JSON document

Holding or rewriting a complete 1.7-million-row document is unnecessary and makes partial validation and diagnostics expensive. Ordered JSONL streams are simpler.

Compress every file based on current size

Dynamic thresholds create format drift and make a harmless row-count increase change file identity. V1 fixes which streams are compressed.

Include segments and crosswalks in the core root

Segments are annotations and legacy IDs are migration evidence. Either could change without changing exact corpus text, so both are excluded from the core component.

Deferred Decisions

• ADR-004: exact token-form and token-occurrence row semantics.
• ADR-005: exact span coordinate and containment representation.
• ADR-006 and F2-019: source/provenance row fields and shared run contracts.
• F3-005: JSON Schemas for every v1 stream and manifest.
• F3-006/F3-009: compiler and importer implementation.
• F3-015: full artifact hosting, retention and mirror policy after size measurement.

Revisit Conditions

• A required exact core fact cannot be represented losslessly in canonical JSON.
• Measured full-corpus size or import time violates release budgets even with fixed zstd streams.
• Two supported runtimes cannot reproduce row, component or archive hashes from the same accepted inputs.
• A future package needs a core stream whose addition cannot be represented by a package-major transition.