ADR-006: Provenance and Source Closure

• Status: Conditional; accepted only when F2-006-W01 is accepted and landed on main
• Date: 2026-08-18
• Roadmap task: F2-006
• Decision owners: provenance, corpus, annotation and release maintainers

Context

Munderecat derives canonical corpus data and annotations from source trees, database snapshots, dictionaries, curated files and human decisions. Copying all raw material into the repository would obscure what the product actually uses, while retaining only final rows would make corrections and clean rebuilds unauditable.

ADR-001 classifies source locators, checksums, runs and validation as control evidence rather than core. ADR-003 requires a hash-bound provenance companion but leaves its row semantics to this decision. ADR-018 requires every public data release to state its rights and attribution basis without implying that repository location grants rights in imported content.

One-directional lineage alone is insufficient. It can prove that an output names a source while still bundling thousands of unused source rows. Minimality alone is also insufficient because a small bundle can omit the evidence needed to reproduce an active output. Publication therefore needs both directions and a closed account of the candidate intake boundary.

Decision

Shared provenance authority

All data domains use one typed provenance graph. Core, compounds, Ottoman renderings, dictionary senses, headings, notes, repetitions, translations, fixtures and migration crosswalks do not invent private lineage ledgers.

The graph is control evidence. It records where a fact came from, what acted on it, and why it is bundled. It does not decide what text reconstructs, which compound is correct, which sense applies or which annotation is current. Those assertions remain in core or their typed domain authority under ADR-001.

Core reconstruction never joins provenance. A reader may continue reconstructing an already accepted core release while provenance display is unavailable, but it reports the evidence subsystem as unavailable. A new release cannot be published, repaired or declared reproducible without complete provenance closure.

Identity-bearing node classes

The logical graph has these node classes. F2-019 will choose physical tables without collapsing their identities.

Node	Identity-bearing referent	Mutable metadata that is not identity
source_system	One named upstream collection or project origin	Display name, homepage, contact and current availability
source_release	One immutable captured state of a source system	Preferred mirror and access status
source_artifact	One artifact occurrence in a source release	Cache path, download URL and local mount
source_record	One exact selected record/range in an artifact	Friendly locator and normalized search key
derived_artifact	One immutable process-output artifact occurrence	Cache path and preferred distribution mirror
process_contract	One versioned acquisition/extraction/transformation/import/validation contract	Documentation label
process_run	One execution or accepted human curation occurrence	Operational display label
assertion	One exact project/human statement used as evidence	Editorial note that does not alter the statement
validation_attestation	One result over exact subjects, policy and validator	Presentation summary
release_manifest	One immutable publication envelope	Current/latest alias and distribution mirrors

Canonical core and domain entities are graph targets, not reissued provenance nodes. Their existing IDs and kinds are referenced through typed endpoint contracts. A raw table_name + record_id pair is forbidden. F2-013 must enumerate referenceable kinds, and F2-019 must provide foreign-key or catalog enforcement for each endpoint kind.

Adding evidence to an unchanged entity preserves that entity’s ID under ADR-002. A published provenance node or manifest is itself immutable; correcting it creates a new node or release envelope and preserves the false or superseded record for audit.

Source system and release

A source_system identifies the logical origin, not its current URL. Examples include one upstream text collection, one dictionary project or the Münderecat curation repository. Renaming a site or moving a repository does not issue another source system ID; changing what collection the ID denotes does.

A source_release freezes one accepted source state and records at least:

• source system ID;
• release-key scheme and immutable source release key;
• capture profile and encoding;
• complete artifact-inventory count and root;
• exact upstream version, commit or release identifier when one exists;
• capture evidence and content digest;
• availability/retention class; and
• rights/attribution classification reference.

An upstream branch name, latest URL, filesystem mtime or query time is not a release key. When upstream supplies no immutable release, the project captures an exact artifact inventory, assigns its own source-release ID and binds that ID permanently to the inventory root. A later recapture is a different release even if its label is the same.

Artifacts, records and selectors

A source_artifact binds one source release, artifact role, safe logical locator, media/encoding profile, exact byte length and cryptographic digest. Directory or repository captures use a canonical path/type/size/hash inventory root; the directory path itself is never identity. Identical bytes occurring in two source releases may reuse content-addressed blob storage, but each release retains its own artifact occurrence and provenance.

Mutable URLs, absolute host paths, Docker volume names, credentials, database object IDs and timestamps are operational locators only. Public records expose a safe stable locator or documented source-local key when useful, but digest and release binding remain authoritative.

A source_record selects an exact portion of one artifact with:

• a versioned selector profile;
• a structured selector value;
• exact selected logical byte count and SHA-256;
• the raw field/value digest set required by that selector profile; and
• optional source-local labels kept only as locators.

Examples are a canonical JSON pointer, a JSONL line identity, a CSV record under a declared parser profile, or a SQLite table/key/column selector against a content-addressed read-only database artifact. A database integer or line number alone never identifies evidence: changing artifact digest, selector profile, selector or selected bytes changes the source-record referent.

The project need not copy full raw artifacts into Git. A retained external artifact may support independently verified selectors when its exact digest, byte count, retention locator class and acquisition contract are recorded. If exact bytes cannot be retained or reacquired under the declared verification class, the artifact cannot serve as an independently verifiable terminal source.

A derived_artifact is an intermediate or distributable artifact emitted by exactly one generating run output edge. Its identity binds generating run ID, output role and position, schema/media/compression profiles, logical and stored byte counts/hashes, and ordered content root where applicable. It may feed any number of later runs. Repeating a run creates another derived-artifact occurrence even when exact bytes are equal; content-addressed blob storage may deduplicate those bytes without merging their provenance identities. A derived artifact never becomes upstream source merely because a later run consumes it.

Assertions and evidence roles

Some facts originate as project-authored or human curation statements rather than as rows in an external file. An assertion records an exact canonical payload, assertion kind, author/actor, creation channel, reason, review context and payload hash. A version-controlled deterministic assertion follows ADR-002 package identity; an interactive assertion uses generated-once identity.

Assertions never masquerade as source bytes. They may justify a boundary choice, homograph decision, manual override, inclusion/exclusion reason or other editorial judgment, but cannot prove that an imported string equals an unavailable source.

Every lineage edge has one controlled role and a compatible typed endpoint pair. V1 roles include:

• direct_source for source content represented by an output;
• transformation_input for a complete process input;
• curation_basis for evidence considered by a human decision;
• boundary_basis for structural selection;
• validation_subject and validation_result;
• rights_basis and attribution_basis;
• fixture_basis and negative_fixture_basis; and
• exclusion_basis for a declared intake disposition.

Generic related_to edges do not satisfy closure. Domain-specific confidence or semantic preference belongs to the domain assertion; provenance records the evidence and decision event without converting evidence quantity into truth.

Process contracts and runs

A process_contract is a reusable, content-addressed recipe. According to its kind, it binds:

• implementation source commit and executable/artifact digest;
• locked runtime and dependency environment identity;
• configuration, parser, normalization and selector profiles;
• input and output schema IDs and hashes;
• ordering, aggregation and error policies;
• required capabilities and resource ceilings; and
• determinism classification.

A process_run is one immutable occurrence under one contract. It binds the complete declared input inventory/root, semantically ordered input edges where order matters, output inventory/root, row counts, executor class and terminal status. Operational timestamps and host identity may be retained privately for audit but do not define data identity or prove order.

Acquisition, extraction, transformation, import, validation and curation are distinct run kinds. Their permissions and semantics cannot be inferred from a generic command name:

• acquisition captures source artifacts without changing them;
• extraction selects source records without asserting canonical meaning;
• transformation generates candidate or canonical-shaped output under a declared contract;
• import validates and publishes an already identified package into staging/DB;
• validation reads exact subjects and emits an attestation; and
• curation consumes evidence and emits an accepted/rejected/superseding domain decision under F2-007.

A failed or interrupted run generates no accepted output edge. Repeating a run creates a new run ID even when deterministic output hashes are equal. A release selects the accepted run/attestation set that justifies it; other run history may remain audit evidence but cannot silently replace the selected set.

Every generating run declares all semantic inputs. Undeclared environment state, network responses, current database contents, locale, wall clock and filesystem order are forbidden inputs. A captured network response first becomes a source artifact; current remote content is never read implicitly during a deterministic rebuild.

Directed graph contract

Lineage direction is:

terminal source/artifact/record/assertion
  -> process run
    -> generated entity/artifact/attestation
      -> later process run
        -> released output

Source-system/release/artifact/record containment is a separate typed hierarchy and must also be acyclic. Lineage edges point from input or evidence to activity and from activity to output. A materialized derived_from shortcut is a replaceable projection of an existing path and cannot introduce authority or omit intervening runs.

The accepted graph has no self edge, no lineage cycle and no edge whose endpoint kind is incompatible with its role. Timestamps do not establish acyclicity. Validation uses graph topology and immutable generation/consumption edges.

An output may have many source records, and one source record may support many outputs. Large releases need not duplicate an edge for every token when a typed coverage binding points to a content-addressed stream/range/set whose membership expands deterministically to exact entity IDs. A mutable SQL predicate, normalized text query or current table scan is not a coverage binding.

Closure boundary

Every candidate publication declares one authoritative private-audit boundary and one disclosure-approved public projection. The private boundary contains:

• release kind and candidate release ID;
• exact core and/or annotation release inputs;
• complete released-output inventory and root;
• source-system/release/artifact/record inventory roots;
• derived-artifact, assertion, process-contract, run and edge inventory roots;
• prerequisite validation-attestation roots produced before closure validation;
• ordered content-addressed historical-manifest dependency inventory and root;
• complete disclosure-approved public-projection inventory and root;
• closure-policy ID and hash;
• rights/attribution policy ID and hash;
• access-policy and public-disclosure-policy IDs and hashes;
• intake partition manifests and roots; and
• known limitations and exclusions.

Both boundaries use typed IDs and canonical ordered inventories, never live database queries. Prerequisite attestations, such as exact reconstruction checks, may be inside the private audit candidate root because they bind already completed subjects. The closure report and its final closure-validation attestation cannot be inside the root they validate. Cross-boundary historical traversal is valid only through an exact dependency in the declared historical-manifest root.

The publication hash sequence is strictly acyclic:

1. The private audit candidate root binds every exact private inventory, historical dependency, public-projection root and policy. It excludes closure reports, final closure attestations, audit/public manifests and registry entries.
2. The canonical private closure report binds the private audit root and public root; its disclosure-approved public report projection binds only the public root. Each report has no embedded self-hash field, and its SHA-256 is computed over complete exact bytes.
3. The final private closure-validation attestation binds both roots and both report hashes. Its public projection exposes only the disclosure-approved result and an opaque audit-attestation reference.
4. The restricted audit manifest binds both roots, both report hashes and the private attestation hash.
5. The distributable public manifest binds only the public root, public report hash, public attestation projection and disclosure-policy hash. It never exposes the private audit root or a guessable private commitment.
6. The configured trusted registry privately binds both manifest hashes, both roots and their attested relationship.

Changing any included inventory, policy, attestation or known limitation creates a new release manifest/bundle identity. It does not change an unchanged core digest or silently mutate the source release.

ADR-003 package projection

ADR-003 companions/provenance-bindings.jsonl.zst is a deterministic projection of this shared graph for one corpus package. It binds each core output ID or exact coverage set to the selected process runs, evidence paths and closure boundary. It does not define a second source catalog or assign new IDs to graph nodes.

The companion may embed the required graph records or reference an externally retained content-addressed source/run manifest only when the outer package binds that manifest’s schema, byte count, hash and retention/verification class. A mutable API or database lookup cannot complete package provenance during import. The importer validates the bound closure before publication, while core reconstruction remains independent of the companion. A companion change alters the ADR-003 bundle identity, not the unchanged core release digest.

A publicly distributed companion contains only the disclosure-approved public graph projection. Private selectors, record witnesses and audit diagnostics remain in the access-controlled audit projection; a public package cannot expose them merely because the authoritative graph contains them. The trusted registry binds both projections under their declared access/disclosure policies without requiring private payload in the public bundle.

Output lineage closure

Output lineage closure answers: Does every published datum have complete support?

Starting from every released entity/row in the declared output inventory, validation walks provenance backward through complete generating-run inputs until every path ends at an allowed terminal:

1. an exact retained or reacquirable source artifact/record under its declared verification class;
2. an exact source artifact/record in an immutable project-owned source release; or
3. an accepted human assertion for an output class that permits editorial judgment.

Project-authored files do not form a fourth implicit terminal type. A tracked curated file, source declaration or fixture is captured under the Münderecat source system and an immutable Git/source release, then addressed as an ordinary source artifact/record. An assertion remains a statement, not byte provenance.

Each domain/output class declares which terminal kinds and evidence roles are permitted. A human assertion cannot replace required direct source evidence for exact corpus characters. A validation attestation proves that a check ran; it is not a source terminal for the checked content.

A generating run is closed only when its complete declared input set, process contract and output membership are present and hash-consistent. Reaching one valid source along one path does not excuse another missing declared input.

The exact output membership of coverage bindings is expanded during validation. Each released output must be covered exactly as its schema requires; duplicate evidence is allowed but does not conceal an uncovered output. The deterministic missing_support set contains typed output ID, required evidence class/role and the first unresolved edge or node.

Used-source closure

Used-source closure answers: Why is every bundled provenance/source item here?

Every bundled source record, derived artifact, assertion, evidence edge and non-structural evidence artifact must reach at least one enumerated use root by a valid forward path. Every root and path belongs to the candidate closure boundary:

1. an entity present in the candidate released-output inventory, including historical lifecycle records only when that candidate intentionally republishes them;
2. a curation decision present in the candidate inventories and explicitly required by the candidate closure/curation policy;
3. candidate release reproducibility metadata required to acquire or verify a used source; or
4. a tracked fixture or negative-test manifest entry bound by the candidate.

Evidence retained only for an earlier public release remains in that immutable historical bundle. A candidate may depend on it without copying its payload only by naming the exact historical manifest as a content-addressed required dependency under its closure policy. An unbound historical entity or merely retained curation row cannot justify otherwise unused evidence in the current bundle.

The reproducibility exception is intentionally narrow. It permits source-system, source-release and artifact-level inventory/retention metadata needed to verify a used source. It does not permit bundling unused raw payload records or an entire dictionary under a generic “reproducibility” label.

A fixture use must name a tracked fixture manifest, test purpose and expected outcome. A negative fixture remains evidence only while that candidate-bound manifest references it. Rejected, retired or superseded decisions retain evidence through the release inventory or historical-manifest dependency that actually preserves them. An unpublished discarded candidate does not justify permanent public bundling unless it is an explicit fixture.

Structural ancestors required to interpret a used record, such as its artifact and source release, inherit that record’s use. All other unconsumed items enter the exact unused_bundled_evidence set and block publication.

Process contracts, runs and validation attestations inherit use only when they occur on a valid retained path to an allowed use root. Merely executing a tool does not justify bundling its run record. Release-wide policy and closure records inherit use from the exact release manifest that binds them.

Intake partition closure

Bulk extractors additionally prove what they considered. Each intake adapter declares an enumeration profile and a bounded candidate-universe manifest. Every discovered candidate identity appears exactly once in one disposition:

• included with the emitted record/output binding; or
• excluded with a controlled reason, rule/curation evidence and optional aggregate group when individual payload retention is unnecessary.

Allowed exclusion reasons include out-of-scope, unused, duplicate, stale, unsafe, unverifiable, rights-blocked and explicit negative-test handling. “Unknown” is not a publishable disposition. Duplicate groups name one retained representative and prove exact membership; normalized equality alone cannot prove duplication.

For a huge external artifact, the candidate universe and exclusion groups may be represented by counts and ordered roots rather than copied raw rows. The adapter must still prove a disjoint, total partition of its declared universe. If an external API cannot enumerate globally, the captured request/query/page boundary is the universe; the release must not claim closure beyond it.

This partition prevents both accidental omission and indiscriminate copying. It is not semantic acceptance: included means the candidate entered the declared next stage, not that its interpretation became canonical.

Closure report and publication gate

One deterministic closure validator emits a machine-readable report containing at least:

• boundary and policy IDs/hashes;
• inventory counts and roots by node/edge kind;
• missing_support;
• unused_bundled_evidence;
• dangling_or_incompatible_edges;
• lineage_cycles and containment_cycles;
• incomplete_runs and hash/count disagreements;
• unverified_terminal_sources;
• intake_partition_gaps, duplicates and overlaps;
• rights_or_attribution_blockers;
• public_metadata_leaks; and
• report schema/version, applicable private/public root and validator artifact identity.

Publication requires every blocking set to be empty and every expected count/root to match. Warnings and known limitations use controlled codes and cannot downgrade a blocking item. Filtering the report, trusting current DB row counts or accepting a sample does not establish closure.

Closure validation runs against one immutable candidate snapshot. It records the private audit and public-projection roots before and after validation and publishes nothing on drift. Private and public report/attestation forms are retained with their respective manifests and registry bindings.

Validation success does not grant publication authority. Every provenance-bearing release applies ADR-003’s trusted-registry rule: publication starts only from an immutable accepted entry in the publisher’s configured trusted registry. That entry binds release ID, private audit root, public-projection root, private/public report and attestation hashes, both final manifest hashes and governing policy hashes. Candidate, package or caller-supplied registry claims cannot authorize publication.

The publication boundary reverifies the accepted entry and acquires uniqueness-backed claims for all release, manifest, provenance and newly issued identity records in one compare-and-publish transaction. It exposes the complete release atomically or exposes nothing; a conflict, missing private-audit commitment, hash disagreement or concurrent winner rolls back every claim and staging change. A registry implementation outside the database supplies an equivalent immutable authenticated decision that is checked inside the visibility transaction.

Rights, attribution, access and privacy

Provenance and redistribution rights are related but distinct. Every public release positively binds a rights and attribution classification for each included source scope, with inheritance only through an immutable source-release/artifact hierarchy. unknown or conflicting classification blocks public distribution of affected payload; a public URL or open repository does not imply permission.

Restricted or private evidence may be retained outside Git. The authoritative private audit projection keeps exact selectors, witnesses, sizes and source identities needed for authorized verification. The public projection is generated separately under the candidate’s content-addressed disclosure policy. It exposes only approved coarse verification/rights classes and safe labels.

Plain hashes, sizes, selectors and source identities can disclose low-entropy private values through guessing or confirmation. The public projection therefore omits them or replaces them with disclosure-approved opaque random IDs or keyed commitments; the key and private mapping remain outside the public manifest. The trusted registry binds the exact private-audit root and public-projection root. The public manifest exposes the public root and an approved opaque audit-attestation reference, not a reversible or guessable private-record commitment.

Neither projection contains credentials, signed URLs, absolute user paths, private hostnames, query secrets, raw personal data or Docker identifiers. Authorized audit resolves private locators through a separate secret boundary. Leak validation scans the exact serialized public manifest, companion, report projection, diagnostics and API fixtures; checking only structured source rows is insufficient.

A release clearly distinguishes:

• independently reproducible public evidence;
• externally available but not redistributed evidence;
• restricted evidence verified by authorized reviewers; and
• project/human assertions.

These classes are not ranked as semantic truth, but clients can accurately state what can be independently reproduced.

Query behavior

Provenance APIs expose typed, release-bound operations:

• trace exact output returns selected runs, evidence roles, terminal sources and verification classes for the requested release;
• why bundled returns the allowed use roots and paths for one bundled item;
• source occurrence returns only projection-approved catalog/release/artifact/ selector metadata without treating the external key as a canonical entity ID; restricted records return opaque public references; and
• closure summary returns inventory roots, blocking-set counts, limitations and the retained report/validator identities.

APIs never resolve through latest implicitly and never infer provenance from equal text, normalized keys or current table contents. Missing retained evidence returns integrity_error or evidence_unavailable according to whether the accepted manifest promised availability; it does not manufacture a replacement source link.

Validation Requirements

Before a provenance-bearing release is publishable, validation proves:

1. Every node and typed edge has a valid stable identity and immutable release binding; mutable locators are not identity inputs.
2. Source hierarchy inventories, artifact bytes, record selectors, selected-value witnesses and derived-artifact outputs match their declared roots and profiles.
3. Process contracts and runs bind complete exact inputs, outputs, schemas, tools and policies; only successful accepted runs generate released outputs.
4. Source containment and lineage graphs are acyclic, endpoint-compatible and free of dangling references.
5. Every released output satisfies its domain’s output lineage closure, including all inputs of every traversed generating run.
6. Every bundled evidence item satisfies used-source closure under one or more candidate-bound enumerated uses or exact historical-manifest dependencies.
7. Every declared intake universe has a total, disjoint include/exclude partition with no unknown disposition.
8. Coverage bindings expand deterministically to exact IDs and agree with output and evidence inventory roots.
9. Rights, attribution, access, verification and disclosure policies are complete; exact serialized public artifacts contain no forbidden or inference-prone private metadata.
10. Private audit/public roots, externally hashed reports, final closure attestations, audit/public manifests and immutable accepted registry entry follow the declared acyclic sequence; atomic compare-and-publish observes no snapshot drift or conflicting identity claim.

Consequences

Benefits

• A contributor can trace a token, compound, Ottoman rendering or sense decision back through exact transformations without needing the legacy database at runtime.
• Minimal bundles are mechanically enforced instead of relying on a cleanup promise.
• Complete intake partitions explain both inclusion and exclusion without copying raw archives into Git.
• Shared primitives prevent domain pipelines from producing incompatible audit ledgers.
• Provenance correction and source movement preserve stable public entity identity and historical releases.

Costs

• Pipelines must emit complete manifests and typed edges rather than informal log lines.
• Exact record selectors and coverage manifests require adapter-specific contracts.
• Retained external artifacts need content-addressed storage and access governance.
• Closure over full releases adds deterministic build work, though range/set coverage avoids occurrence-by-occurrence duplication.

Alternatives Considered

Copy every raw source into the repository

This makes discovery easy but violates minimality, increases legal/operational risk and obscures which records affect the product. Content-addressed external retention plus exact selectors provides traceability without a raw archive dump in Git.

Store one source URL on each final row

URLs move, do not identify versions and cannot express multi-input transformations or human decisions. Typed source releases, records, runs and edges are required.

Keep only transformation logs

Logs are operational narratives, not complete immutable input/output contracts. They cannot prove graph closure or deterministic membership.

Require direct source edges on every token occurrence

This is mechanically simple but needlessly duplicates millions of edges. Exact content-addressed coverage sets retain per-entity traceability as a derived expansion.

Treat a closure warning as sufficient

Warnings would allow missing support and unused source payload to accumulate until the release becomes unauditable. Blocking sets remain fail closed; controlled known limitations cannot contain closure failures.

Let each domain define provenance independently

This would duplicate source releases, transformations, rights facts and closure rules. Domain semantics remain typed, but provenance primitives and publication gate are shared.

Deferred Decisions

• F2-007: production curation state, actor authorization and decision transactions.
• F2-010: public API envelopes, authorization and RLS behavior.
• F2-013: referenceable entity catalog and typed target enforcement.
• F2-017/F2-018: release signatures, migration ledger and compatibility impact.
• F2-019: physical source/run/evidence/curation schemas and canonical serialization.
• F3/F5: concrete source adapters, coverage manifests and full-corpus closure reports.

Revisit Conditions

• SHA-256 or a selected selector/canonicalization profile is no longer adequate for the retained source class.
• A required source legally or operationally cannot be retained or verified under any declared access class.
• Exact coverage expansion costs exceed release budgets and no content-addressed range proof can preserve per-entity traceability.
• A domain demonstrates that the shared graph cannot represent necessary evidence without granting provenance semantic authority.