Machine models that fill gaps in damaged ancient texts learn from scholarly editions. A large share of the "text" in those editions was never read on the stone or papyrus. It is an earlier editor's guess, printed in brackets. The Machine-Actionable Ancient Text corpus alone contains about 750,000 such expert restorations (A new machine-actionable corpus for ancient text restoration). When a model trains on guesses and is then scored against guesses, it becomes the next link in a chain of learners passing knowledge down generations. Work on intergenerational transmission makes a precise prediction about such chains. Iterated learning: Intergenerational knowledge transmission reveals inductive biases shows that when each learner infers from the previous learner's output and then produces new data, the chain forgets its starting point and settles on the learners' shared prior. In their experiment, 28 of 32 human chains collapsed to a positive linear function within three to five generations, whatever function they started from. Applied to philology, this predicts that restored text will be more formulaic than the text it replaces, that rare words and names will disappear from restorations first, and that each new round of model-assisted editing will speed this up.
This note proposes to test that prediction directly and to build restoration training and evaluation that account for it. The two-sentence version is as follows. Ancient-text restoration is itself an intergenerational transmission chain, because models learn from earlier editors' conjectures. Iterated-learning theory therefore predicts, and we will measure, a drift toward common formulas that explains the known frequency bias and rare-word failures of current restorers. We then test whether tracking which tokens are attested and which are conjectured, during both training and evaluation, reverses that drift.
Core Claim
Current restorers fail on exactly the tokens a prior-seeking chain would erase. On the EvaCun cuneiform benchmark, token prediction peaks around 21% and out-of-vocabulary accuracy is "effectively 0%" (EvaCun 2025 Shared Task). Fine-tuned LLMs systematically overpredict common administrative words and underpredict rare terms (Finetuning LLMs for EvaCun 2025 token prediction shared task). A Joseon-annals restorer with 89.09% HITS@10 overall falls to 8.3% on proper nouns (Restoring and Mining the Records of the Joseon Dynasty).
Prior work reads these as capacity or data-sparsity problems. Our claim is that part of the failure is structural. Training targets are contaminated by earlier conjectures, and the contamination is biased toward the conjecturers' priors. The claim also makes a sharper prediction. Iterated learning distinguishes learners who sample from their posterior, whose chains converge to the prior, from learners who pick the single most probable answer, whose chains overshoot toward the prior's mode. So the common practice of majority-vote ensembling, which gave the best EvaCun number (26.9%, Jon & Bojar), should accelerate the erasure of rare forms even while it raises top-1 accuracy.
A simple way to state the mechanism is to write the effective training target in gap-like contexts as
$$p_{\text{eff}} = (1-\rho)\, p_{\text{true}} + \rho\, q_{\text{prev}},$$
where $p_{\text{true}}$ is the distribution of what scribes actually wrote, $q_{\text{prev}}$ is the distribution of earlier restorers' conjectures, and $\rho$ is the fraction of training tokens that are conjectures. Suppose $q_{\text{prev}}$ is itself produced by a model trained on the previous $p_{\text{eff}}$. Then repeating this mixture is the iterated-learning Markov chain, and its fixed point is pulled toward the restorer's prior at a rate governed by $\rho$. Nothing here is new mathematics. What is new is the observation that $\rho$ is large, measurable, and ignored in this field.
Relation to Prior Work
Two strands of work motivate this project. The restoration papers provide the failure pattern: the EvaCun shared task and Jon & Bojar for cuneiform, Kang et al. for Hanja, and the Babylonian LSTM of Fetaya et al., which reached 85% Hit@1 on formulaic Late Babylonian archival texts. All of them treat every token in an edition as equally observed. None separates bracketed conjecture from attested reading.
The transmission papers supply the mechanism. Griffiths et al. give the convergence-to-prior theorem. Overcoming Individual Limitations Through Distributed Computation shows that chains which sample beliefs and check them against fresh evidence aggregate information rationally, while naive copying helps only when evidence is strong. That result is the reason our fix re-anchors training on attested evidence rather than on consensus.
Emergence of Compositional Language with Deep Generational Transmission acts as a constraint. It shows that generational bottlenecks strip "idiosyncratic" forms, and that doing so raised novel-pair accuracy from 13% to 46%. That is desirable for emergent communication and harmful for philology, where the idiosyncrasies are the historical record. We take this as evidence that bottleneck filtering is real in neural learners, not only in humans.
Critical Confabulation is the nearest cross-over, since it fills archival silences with an LLM. It treats the silence as a static gap in a fixed archive and does not model how the archive's filled-in content was produced. We do the reverse and model the production of filled-in content itself.
Research Program
The first component is a descriptive test on human editors, which needs no model training. MAAT explicitly marks restored spans and gap lengths and records alternative readings from the critical apparatus. For each gap-length bucket and language (Greek, Latin), we compare restored tokens with attested tokens drawn from matched contexts. The comparison covers token entropy, mean corpus frequency, the proportion of hapax and low-frequency types, and the proportion of proper names. Iterated learning predicts that restored text is lower-entropy and more frequent-word-heavy than attested text of the same length and genre. For cuneiform we will parse eBL and EvaCun transliterations with the ebl-atf tooling described in Transliterated Cuneiform Tablets of the Electronic Babylonian Library Platform. ATF conventionally brackets broken and restored signs. We will verify that the markup survives in the exports; if it does not, the cuneiform arm is limited to Achemenet. A null result here, meaning human restorations are statistically indistinguishable from attested text, would weaken the whole thesis early and cheaply.
The second component is a provenance-split evaluation. Every masked test position is labeled "attested-mask" (a truly read token that we hide) or "restoration-mask" (a position whose gold label is an earlier conjecture). Models are scored separately on each. We also report rare-token recall on the bottom frequency decile and on proper nouns, out-of-vocabulary recall, and the number of distinct types a model predicts across the test set. Our prediction is that current methods look best on restoration-mask positions and worst on rare attested tokens. That gap would be direct evidence that the benchmarks reward agreement with the scholarly prior.
The third component is a controlled restoration chain. We start from the attested-only portion of a corpus and hide real gaps, using MAAT's own empirical gap-length distribution. We train a restorer, let it fill the gaps, retrain a fresh model on the filled corpus, and repeat for about six generations. We run three arms: sampled restorations, argmax restorations, and majority-vote ensemble restorations. After each generation we track the number of distinct types used in filled positions and the divergence of the filled-token distribution from held-out attested text. We also track the survival rate of rare types, meaning those present in the generation-0 gold that are ever produced in generation $k$. Theory predicts extinction ordered argmax ≥ ensemble > sampling, with sampling converging to the prior rather than to the truth.
The fourth component is the intervention: provenance-aware training. Each token carries a provenance tag, attested or conjectural, both as an input embedding and in the loss. Conjectural targets are down-weighted. Where the apparatus offers alternative readings, they are replaced by a soft target spread over those alternatives. Training also includes a steady stream of attested-mask examples, which plays the role of the "fresh evidence" that keeps the social-sampling populations of Hardy et al. rational. At inference, the model reports a distribution over candidates rather than a single token, and we evaluate that distribution for calibration.
Baselines
The naive reference points are a most-frequent-token lookup and a 2-gram model, as in Fetaya et al., where the 2-gram reached perplexity 22.87 against 5.02 for the LSTM. They set the floor and show how much of any score comes from pure formula frequency. Any method that fails to beat the 2-gram on rare attested tokens has learned nothing beyond the prior.
The strongest existing method is QLoRA fine-tuning of an open LLM with the "All" prompt, which predicts every masked token at once, plus majority voting over top checkpoints. Jon & Bojar report 26.9% accuracy for this ensemble on the EvaCun development set (single model up to 22.1%), and the shared task's best official system reached 21% (EvaCun 2025). We will reproduce it on EvaCun with the same model family (Mistral Small 3 24B or Aya Expanse 8B), the same training tokens, and the same number of checkpoints. We will match GPU-hours and the hyperparameter-search budget across all arms. On MAAT the same recipe runs on a Greek-capable open model; no reported result found on Lacuna for any trained model on MAAT, so there we establish rather than beat a number.
Beating this baseline means one of two things. The first is a clear gain in rare-token and proper-noun recall on attested-mask positions, with top-1 accuracy on the full test set within one point of the ensemble's 26.9%. The second is a slower type-extinction curve in the chain experiment. Gains only on restoration-mask positions would not count.
Two ablations isolate the key ingredient. The first is "drop conjectures", which removes all bracketed tokens from training targets; this tests whether provenance tagging adds anything beyond data cleaning. The second is "prior-corrected decoding", which subtracts a scaled log unigram frequency from the baseline's scores at inference. It tests whether a one-line recalibration captures the effect without any retraining. A third, internal ablation removes the apparatus soft targets but keeps the down-weighting, to show whether the alternative readings carry information.
Why not just…?
Why not just drop the restored spans from training? This is a real threat, and it is why dropping is a full experimental arm rather than a straw man. Dropping discards about 750,000 expert judgments in MAAT, which make up 83% of Greek restorations. It also removes the contexts around gaps, which are exactly the damaged regions the model must handle at test time. If dropping matches provenance-aware training on attested-mask rare-token recall, we will report that as the main practical result. The diagnostic contributions, the provenance-split benchmark and the chain dynamics, would still stand, because they are what tell the field that dropping is necessary.
Why not just correct the frequency bias at decoding time? Subtracting a unigram prior addresses the symptom Jon & Bojar describe and costs nothing, so we test it. Iterated-learning theory suggests it will be only partly effective. The contaminated prior is contextual, favoring formulas in specific positions, rather than a global word-frequency skew. A global correction will also hurt the high-frequency tokens that are actually correct. The provenance-split evaluation will show whether the correction recovers rare attested tokens or merely trades accuracy on common tokens for noise.
Why not just use bigger models or larger ensembles? Scale does not appear to be the binding constraint. Moving from 8B to 34B models in Jon & Bojar did not break the low-20s ceiling, and out-of-vocabulary accuracy stayed near zero across the shared task. Our chain experiment also predicts that majority-vote ensembling specifically makes the problem worse, because it acts as an argmax learner. If ensembles instead preserve rare types in the chain, that falsifies our account.
Why not just replace exact match with a softer semantic metric? Rigid exact-match scoring is a known complaint in EvaCun and MAAT, and softer metrics will be reported alongside exact match. They do not address the issue here, though. A metric that is lenient toward plausible alternatives rewards prior-consistent guesses even more. The question is whether the gold label is itself a conjecture, and only provenance splitting answers that.
Decisive Checks
Three results would confirm the proposal. First, human restorations in MAAT are measurably more formulaic than matched attested text. Second, current restorers score higher on restoration-mask positions than on attested-mask positions, with the deficit concentrated on rare types and names. Third, in the chain experiment, argmax and ensemble restorers drive rare types extinct faster than sampling restorers, and provenance-aware training slows extinction relative to both "drop conjectures" and prior-corrected decoding. If the first and second fail, restoration contamination is not a real driver of current failures and the thesis should be abandoned. If the third holds but the intervention does not beat "drop conjectures", the contribution narrows to a benchmark and a warning. That warning becomes more urgent as model outputs begin to enter published editions.
References
- A new machine-actionable corpus for ancient text restoration
- Iterated learning: Intergenerational knowledge transmission reveals inductive biases
- EvaCun 2025 Shared Task: Lemmatization and Token Prediction in Akkadian and Sumerian using LLMs
- Finetuning LLMs for EvaCun 2025 token prediction shared task
- Restoring and Mining the Records of the Joseon Dynasty via Neural Language Modeling and Machine Translation
- Restoration of fragmentary Babylonian texts using recurrent neural networks
- Overcoming Individual Limitations Through Distributed Computation: Rational Information Accumulation in Multigenerational Populations
- Emergence of Compositional Language with Deep Generational Transmission
- Critical Confabulation: Can LLMs Hallucinate for Social Good?
- Transliterated Cuneiform Tablets of the Electronic Babylonian Library Platform