ADR-004: Token Form vs Token Occurrence

• Status: Conditional; accepted only when F2-004-W01 is accepted and landed on main
• Date: 2026-08-18
• Roadmap task: F2-004
• Decision owners: corpus engineering and schema maintainers

Context

The accepted legacy corpus has 152,786 deduplicated tokens rows and 1,696,548 ordered sentence_tokens rows. The old token catalog also contains default punctuation, whitespace, normalized text, roots and POS guesses. Those values mix four different concerns: exact writing, one source occurrence, linguistic analysis and display convenience.

Munderecat needs stable occurrence anchors for spans, citations, compounds and “where used” queries while preserving every source codepoint. It also needs exact surface reuse without claiming that equal spelling means equal lemma, root, sense or compound role. Runtime tokenization or inferred spaces would move anchors and break ADR-001 reconstruction and ADR-002 public identity.

ADR-003 fixes one compressed token-occurrences.jsonl.zst core stream but leaves its row semantics to this decision. ADR-005 will define span coordinates over the occurrences fixed here.

Decision

Canonical vocabulary

Munderecat distinguishes three concepts:

• A token form is one exact, non-empty orthographic body in one text version. It is a core value identity, not a linguistic analysis.
• A token occurrence is one immutable, ordered body occurrence inside one sentence. It is the public core anchor used by later spans and references.
• A tokenization profile is a versioned, content-addressed semantic contract that identifies form-body intervals and assigns every remaining source codepoint to an exact reconstruction field.

An occurrence has one of two kinds:

• form: references a token form and renders that form’s exact body;
• literal: carries exact non-empty content for a source unit that the profile does not classify as a reusable form, such as a punctuation-only separator sentence.

literal is not a fallback for tokenizer uncertainty. The profile declares where it is valid, and the compiler fails on content it cannot classify deterministically.

Token form boundary

A token form owns only:

• stable public form ID;
• text-version scope; and
• exact, unnormalized Unicode scalar sequence of the token body.

The following never belong to token form identity or its core row:

• leading or trailing punctuation;
• whitespace or a default separator;
• case-folded, ASCII-folded or search-normalized values;
• lemma, root, POS, grammatical role, pronunciation or contextual meaning;
• emphasis, highlight, font or reader presentation state;
• dictionary senses, Ottoman forms or compound membership; and
• frequency, first/last occurrence or other derived statistics.

The same exact body within one text version has one form ID in a core release. The same bytes in different text versions have distinct form IDs; corpus-wide exact search may group them by byte equality without claiming shared identity. Different case, apostrophe, diacritic, normalization form or codepoint sequence is a different token form even when a search UI may later group it.

A form exists in core only when at least one accepted occurrence uses it. Lexicon or dictionary forms that do not occur in a text version belong to domain authority, not this catalog.

Exact equality and deduplication

Form equality is the exact UTF-8 encoding of the accepted, unnormalized scalar sequence. Locale collation, database default collation, Unicode normalization, case-folding, punctuation folding, fuzzy matching and legacy integer IDs cannot deduplicate forms.

Implementations use a binary SHA-256 key for indexing and compare exact bytes before accepting equality. A digest collision is a hard failure, not permission to merge different bodies. Database uniqueness must use deterministic binary equality or the digest-plus-exact-value pair; locale-sensitive unique indexes are insufficient.

This equality defines an orthographic value only. Equal forms such as O at two locations do not imply equal referent or sense. Homograph analyses remain separate annotation assertions and can select different lexemes per occurrence.

Lossless occurrence payload

Every sentence owns one exact initial_text value followed by a contiguous sequence of token occurrences at zero-based positions. The canonical rendering unit for an occurrence is:

body(occurrence) + trailing_text

For a form occurrence, body is the referenced token form’s exact text. For a literal occurrence, body is its exact literal_text. trailing_text is the possibly empty exact sequence from the end of that body to the beginning of the next body, or to the sentence end for the last occurrence. initial_text is the possibly empty sequence before the first body.

Sentence reconstruction is exactly:

initial_text + concat(body(occurrence) + trailing_text in position order)

No renderer inserts a space, punctuation mark, line break or other default. Every accepted sentence codepoint appears in exactly one of initial_text, a form body, a literal body or one occurrence’s trailing_text. The fields never overlap and leave no gap.

An accepted non-empty sentence has at least one occurrence. If the tokenization profile finds no reusable form body, the sentence has one literal occurrence whose body carries the complete non-initial content. Empty source sentences are forbidden by the corpus source contract rather than represented by a synthetic token.

This ownership rule deliberately attaches between-body punctuation and whitespace to the preceding occurrence’s trailing_text. It prevents two valid serializations of the same sentence. Punctuation before the first body belongs to initial_text.

Discriminated occurrence rules

The logical occurrence row has these required semantics even if later SQL names differ:

• public occurrence ID;
• sentence ID;
• contiguous zero-based position;
• occurrence kind;
• token form ID and exact form body for form, otherwise absent;
• exact non-empty literal body for literal, otherwise absent;
• exact possibly empty trailing_text; and
• row hash required by ADR-003.

The form and literal body channels are exclusive and exhaustive. A form occurrence cannot override the form text locally. A literal occurrence cannot carry a form ID. Neither kind can have an empty body.

The canonical package repeats exact form body beside form_id in each form-bearing occurrence row. The importer validates that:

• one form ID always names one text-version/body pair;
• one text-version/body pair always names one form ID; and
• each form ID has at least one occurrence.

It then may normalize the unique pairs into a physical token_forms table and keep only a foreign key on database occurrence rows. This prescribed projection does not add a new ADR-003 stream and does not make token forms a disposable cache: their IDs and exact values remain hash-bound in the canonical occurrence stream.

Ordering and containment

Occurrence order is a source fact, never a query result. For every sentence:

• positions are unique and exactly 0..count-1;
• every occurrence has that sentence as its only parent;
• an occurrence cannot be shared by two sentences or text versions;
• traversal follows text-version, book, chapter, paragraph, sentence and occurrence source positions; and
• UUID order, form order, insertion order and database plan order cannot affect text order.

The first package contains exactly 1,696,548 canonical token occurrences and retains a bijective typed crosswalk to all 1,696,548 accepted legacy sentence_tokens occurrences. Each legacy occurrence has exactly one canonical target and every canonical occurrence is the target of exactly one legacy occurrence; missing, duplicate, many-to-one and unclaimed canonical rows are migration failures. Legacy token catalog rows may map many-to-one to a new exact form because their punctuation/whitespace defaults are not form identity.

If source evidence proves that an accepted legacy token boundary itself is wrong, that correction cannot be hidden inside crosswalk import. It follows the explicit successor core release and occurrence supersession process.

Tokenization profile and release identity

Each text version binds one tokenization profile ID and SHA-256 in its core release envelope. The profile fixes at least:

• scalar-level scanning and body-boundary rules;
• apostrophe, hyphen, digit, abbreviation and mixed-script behavior;
• punctuation-only and separator literal behavior;
• residual-codepoint attachment to initial_text or trailing_text;
• deterministic error behavior; and
• fixture and compiler compatibility version.

The semantic profile ID is included in the token occurrence schema/row contract and therefore in the ADR-003 core root. Refactoring the compiler without changing the profile or canonical rows changes control evidence only. Changing profile semantics, body boundaries, sentence ownership, positions or rendered fields creates a new core release and new affected occurrence identities under ADR-002.

An unchanged form in a successor release preserves its already-issued form ID when the accepted predecessor mapping proves the same text version referent and exact body. An unchanged occurrence preserves its ID only when its governing per-kind identity contract proves the same release lineage, sentence referent, position, boundary and exact rendered unit. Otherwise a successor is issued; IDs are never recomputed from current table order or normalized text.

For the initial deterministic package, occurrence identity paths use stable source coordinates and the declared position. Form identity paths use a deterministic pre-publication catalog coordinate assigned by first source occurrence under the text version, not the form’s display text or a legacy token ID. Later packages preserve issued IDs through accepted predecessor/crosswalk evidence.

Reconstruction and integrity checks

Before a core package is publishable, validation proves:

1. Every occurrence row satisfies the discriminated form/literal rules.
2. Every sentence has one contiguous occurrence sequence and no cross-parent row.
3. Form ID/body mappings are bijective within each text version and use binary exact equality.
4. The concatenated occurrence representation equals the accepted sentence scalar sequence exactly.
5. Sentence assembly produces the expected paragraph, chapter and release bytes and ADR-003 reconstruction digests.
6. Re-tokenizing accepted sentence bytes with the bound profile produces the same positions, kinds, bodies and trailing fields.
7. The initial package has exactly 1,696,548 canonical occurrences; all accepted legacy sentence-token occurrences have one target, every canonical target has one legacy source and neither side contains an unpaired or multiply paired row.
8. Repeating the compiler produces byte-identical occurrence rows, form projection, IDs and roots.

A stored sentence text or digest is an independent comparison authority, not a shortcut that permits malformed occurrence rows. Reconstruction must use only core sentence/occurrence/form facts and cannot read annotation, search or lexical tables.

Search, hover and future lexical layers

The separation creates explicit query modes:

• this occurrence addresses one token occurrence ID;
• this exact form in this text version follows its token form ID;
• same exact bytes across versions is an explicit binary-value query;
• same lexeme/root/sense follows accepted lexical annotations; and
• same compound family follows accepted compound authority and occurrences.

These modes cannot silently fall into one another. A missing lexical meaning does not make exact-form occurrence lookup incomplete. Stopwords such as ve, short forms, abbreviations and Arabic-script forms retain ordinary core form/occurrence identity even when dictionary or Ottoman mapping policy marks them unavailable or not applicable.

A normalized search key may index several exact forms, but it is derived and cannot be returned as the form’s canonical spelling. A lexeme/root correction changes annotation authority and selection, not core token form or occurrence rows.

Compounds and segments select occurrence spans under ADR-005. They never replace, merge or split the core occurrences they cover. A compound such as lisan-i hal therefore owns semantic definition and occurrence-annotation identities while its member token occurrences remain independently referenceable.

Legacy migration boundary

The legacy database is evidence and a crosswalk source, not the new authority:

• tokens.text may seed exact body candidates after source verification;
• tokens.default_whitespace_after, tokens.default_punctuation, normalized text, root and POS do not enter the core form projection;
• sentence_tokens.leading_punctuation, punctuation, whitespace_after and sentence ending punctuation help reproduce old boundaries but are accepted only when their assembly equals source text exactly;
• styling, contextual meaning and notes are separately classified annotation or control evidence; and
• legacy integer IDs remain only in the ADR-003 crosswalk companion.

The canonical source and bound tokenization profile decide disagreements. No import uses a legacy default to fill a missing separator, and no normalized comparison can turn a reconstruction mismatch into success.

Consequences

Benefits

• Exact text reconstruction no longer depends on token-catalog defaults or renderer spacing heuristics.
• Stable occurrence IDs support spans, citations and exact “where used” results.
• Form deduplication reduces storage without collapsing occurrence context, homographs or linguistic interpretation.
• The package keeps ADR-003’s single large token stream while allowing a normalized relational token_forms -> token_occurrences implementation.
• Lexical, compound and Ottoman systems can improve without mutating core token anchors.

Costs

• Canonical package rows repeat form body text for independently verifiable streaming import; zstd mitigates transport duplication.
• Exact binary forms produce more variants than normalized search catalogs and need derived grouping indexes for user search.
• Boundary corrections are expensive because they create a successor core release and require span-impact analysis.
• Physical schemas must enforce cross-row form/body bijection and cannot rely only on a locale-collated text unique constraint.

Alternatives Considered

Keep the legacy token table unchanged

Legacy default punctuation/whitespace and mutable lexical guesses make one shared row influence unrelated occurrences. This violates exact ownership and was rejected.

Store punctuation and whitespace in token forms

This would create separate form identities for layout context and make exact word lookup depend on neighboring punctuation. Punctuation and separators remain occurrence-owned core content instead.

Store only complete occurrence strings

Full strings reconstruct easily but provide no stable exact-form identity and force every lookup to regroup occurrences. The selected package row carries both exact body and form ID while the database may normalize them.

Derive occurrences from sentence text at runtime

Tokenizer upgrades would move public anchors and make citations release-dependent at query time. Tokenization occurs once under a bound profile and publishes immutable occurrences.

Normalize forms before deduplication

Case, apostrophe and diacritic changes can be source-significant and can distinguish valid Ottoman-era spellings. Normalized keys remain derived search aids only.

Create token forms for punctuation and whitespace

This keeps one uniform foreign key but pollutes exact lexical lookup and grants dictionary-like identity to layout fragments. Literal occurrences and exact trailing content preserve them without that conflation.

Deferred Decisions

• ADR-005: span endpoint, offset and containment coordinates over occurrences.
• F2-006/F2-019: source, tokenization-run and transformation evidence fields.
• F2-013: final catalog of which form and occurrence kinds are public reference targets.
• F2-016: canonical glossary names across token, lexeme, form and occurrence models.
• F3-005/F3-006: JSON Schemas, compiler representation and executable tokenizer profile.
• F3-007 onward: physical tables, constraints, import and reconstruction functions.
• F5: lexeme/root/sense, compound, Ottoman and occurrence lookup implementations.

Revisit Conditions

• The accepted source contains a sentence that cannot be partitioned by the form/ literal contract without loss or ambiguous ownership.
• The initial 1,696,548 occurrence crosswalk cannot be bijective despite exact source and reconstruction agreement.
• A required span cannot address punctuation or subtoken content under ADR-005 without changing occurrence identity.
• Two supported runtimes cannot reproduce exact form equality, tokenization rows or reconstruction roots from the same profile and source.