Twin Language Families: Separating Memorization from Reconstruction in LLMs

Large language models now perform well on classical languages, often with no task-specific training. A recent study found that GPT-4o reaches 0.55 Macro F1 on Latin named-entity recognition with no examples, edging past a fine-tuned LatinBERT at 0.54 (A Case Study of Cross-Lingual Zero-Shot Generalization for Classical Languages in LLMs). The authors themselves concede that data contamination is an unresolved limitation: the model may have seen the test items during pre-training. The problem is sharper in historical linguistics than almost anywhere else. The surviving record of an ancient language is fixed and small, and every Latin word, Romance cognate table and etymological dictionary entry has been on the web for decades. So "evaluate on data created after the training cutoff" is unavailable: no new Latin is being written.

We propose to measure how much of an LLM's skill at protoform reconstruction comes from recalling published etymologies and how much comes from applying the comparative method. Protoform reconstruction means inferring an unattested ancestor word, such as a Latin form, from its descendant words in daughter languages. The tool is a generator of twin language families. These are artificial families whose ancestor words are freshly sampled, so no web text contains them. They are matched to a real family in phonotactics, meaning which sound sequences are allowed. They are also matched in the number and kind of sound changes and in how many sounds merge or are lost. Comparing a model on a real family and its twin, and subtracting the same comparison for a small model trained from scratch that cannot have memorized the web, gives an estimate of the advantage attributable to memorized items.

Core Claim

A drop in score when real data is swapped for synthetic data is not, by itself, evidence of contamination. The synthetic data may simply be harder, or less phonetically natural. A²RBench's symbolic remapping test replaces familiar tokens with arbitrary symbols and reports a 17.7% accuracy drop for frontier models (A²RBench). Applied to sound change, that test would also destroy legitimate knowledge, for example that /k/ tends to become /tʃ/ before front vowels. A model that knows real phonetics should do worse on arbitrary symbols.

Our claim is that a ladder of conditions removes one source of advantage at a time. The ladder runs from real words under real sound laws, to novel words under real laws, to novel words under novel but natural laws, to the same with symbols remapped. Paired with a from-scratch model that calibrates difficulty, the ladder separates three things. The first is item memorization. The second is knowledge of the specific documented sound laws of a family. The third is a general phonetic-naturalness prior. Only the first is contamination in the harmful sense.

Research Program

The generator has three parts.

Ancestor sampling. A phonotactic model is fit to the real protolanguage: Latin for the Romance family, Pre-Latin Proto-Italic for PILA, Middle Chinese for WikiHan. New ancestor words are sampled from it. Any sample that matches an attested lexical item, or lies within a small edit distance of one, is rejected.

Forward sound-law cascades. These are executable ordered rewrite rules that turn ancestors into daughter words. Two kinds are used. Real-law cascades are hand-implemented for well-documented changes, such as Proto-Italic to Latin, and a subset of Latin to Spanish and Italian. We accept a cascade only if it reproduces held-out attested reflexes from PILA or Rom-phon at a preset accuracy. A reflex is a descendant word in a daughter language. Novel-law cascades are sampled from a typological inventory of common changes, such as lenition, palatalization, vowel mergers and final-consonant loss. Each sampled cascade matches its real counterpart in rule count, number of mergers and number of losses.

Verification. A²RBench accepts a task only if the inverse transformation undoes the forward one exactly (A²RBench). That condition cannot hold here, because mergers make reconstruction many-to-one. We replace it with a set-valued inverse. The generator enumerates every ancestor that the cascades map to the observed daughter set. A prediction counts as correct if it falls in that set, and we also report accuracy against the single sampled ancestor.

This verification step builds on the bidirectional reranking of Improved Neural Protoform Reconstruction via Reflex Prediction. That work uses a learned reflex predictor as a heuristic to rescore candidate ancestors. We use exact forward rules as ground-truth verification. This is only possible because we own the generative process. That paper's note that wrong candidates often look superficially like their reflexes motivates scoring against the consistent set rather than by surface similarity.

Evaluation ladder. Each real benchmark gets four conditions:

  • C0: real cognate sets.
  • C1: novel ancestors passed through real-law cascades.
  • C2: novel ancestors passed through matched novel-law cascades.
  • C3: C2 with a consistent random remapping of phoneme symbols.

LLMs receive the same number of in-context training cognate sets in every condition, then a query set. For C0 we also run a closed-book variant. The contamination estimate is a difference-in-differences: the LLM's C0−C1 gap minus the from-scratch model's C0−C1 gap. The C1−C2 gap measures reliance on the documented laws of a family, which is textbook knowledge rather than item recall. The C2−C3 gap measures the naturalness prior.

Secondary question on data scarcity. The same generator yields unlimited training families. We test whether pre-training a small reconstruction model on matched synthetic families improves it on WikiHan, which has only 804 cognate sets. This is secondary and reported regardless of outcome.

Baselines

Naive reference. The first reference point is a copy baseline that outputs one daughter word as the ancestor. On PILA it scores a phoneme error rate (PER) of 0.53 (PILA). It anchors each condition's floor and should stay roughly constant across C0 to C2 if the twins are well matched.

From-scratch Transformer reconstructor. The second baseline is the Transformer of Transformed Protoform Reconstruction. It is trained per condition on that condition's training split only, with identical splits and hyperparameter search budget. It reports 71.23% exact match on Romance orthographic data, against 69.80% for the prior recurrent model. On PILA, a comparable Transformer reaches 0.18 PER (PILA). This model is the key competitor because it cannot benefit from web memorization. Its C0−C1 gap is therefore our measure of residual difficulty mismatch. If the gap exceeds a few points of exact match, the twins are not matched and we refit the generator before any LLM result is interpreted.

Reflex-reranked model. The third baseline is the reranked system of Improved Neural Protoform Reconstruction via Reflex Prediction, at 57.26% versus 55.58% accuracy on WikiHan. It is the strongest published reconstruction method in the file, and it is run under the same per-condition protocol.

Frontier LLMs. The LLM subjects follow Akavarapu et al.: a large and a small open model, plus one proprietary model, with identical prompts and the same in-context example count across conditions. No reported result was found on Lacuna for LLMs on Rom-phon, WikiHan or PILA reconstruction, so C0 numbers will be new.

Key ablation. Plain symbolic remapping of real data, which is C0 remapped, isolates whether our matched-naturalness twins add anything over the A²RBench test.

"Beating" the field here means a validation result: the from-scratch model's C0−C1 gap is near zero, while the LLM's estimate has a confidence interval excluding zero, or tightly bounded at zero.

Why not just…?

Why not just run n-gram or MinHash overlap checks? Proprietary training corpora are unavailable (An Open-Source Data Contamination Report for Large Language Models). Overlap filters miss paraphrased and translated test material (A Survey on Data Contamination for Large Language Models; Investigating Data Contamination for Pre-training Language Models). More fundamentally, the answer to every Romance reconstruction item is a Latin word that certainly appears on the web. Overlap is close to 100% by construction and says nothing about whether recall drives the answer.

Why not just measure perplexity or membership on the test items? Low surprise on memorized puzzles is informative in reasoning tasks (Studying and improving reasoning in humans and machines). Here, though, membership is again saturated. It also detects exposure, not whether exposure is what produces the correct reconstruction. We still use it as a covariate: if memorization matters, the C0−C1 gap should grow with Latin word frequency, as frequency effects predict (Snoopy).

Why not just remap symbols on the real benchmarks? This is the cheapest serious alternative, and it is included as an ablation. Its weakness is that it removes item memory and the phonetic-naturalness prior at once. If C0-remapped drops by roughly as much as C0→C1, and C2→C3 is small, then remapping alone suffices and the twin generator is unnecessary. We will report that outcome as such.

Why not just evaluate on post-cutoff data? The historical record does not grow. Newly published etymologies are too few and too contested to form a test set.

Proposed Experiments

We build twins for Rom-phon, Rom-orth, WikiHan and PILA, with 5 sampled twin families per benchmark to estimate variance from the generator. All conditions are scored by exact match, phoneme edit distance and consistent-set accuracy.

Validation check. The from-scratch Transformer's C0−C1 and C1−C2 gaps must be small. Without this, nothing downstream is interpretable.

Main experiment. We estimate the LLM contamination gap per benchmark and per model size. We also test whether the small-model advantage reported in Akavarapu et al. holds on C1 and C2 or shrinks there. This bears directly on the tension with the monolingual-beats-multilingual finding of Exploring Large Language Models for Classical Philology. If large models' lead over specialized models largely disappears on twins, their lead on real data was largely recall.

Error analysis. On C2 we test whether LLMs produce phonotactically illegal outputs, the paradox reported in Probing Multilingual Cognate Prediction Models. Because we know every rule, we can attribute errors to specific mergers and losses.

Data-scarcity experiment. We pre-train the Transformer on 10^4 synthetic families before fine-tuning on WikiHan, and compare against the reported 57.26% reranked accuracy.