Borrowed Words as Shortcuts: Auditing and Repairing Borrowing Contamination in Protoword Reconstruction

Neural models now reconstruct ancestral words, such as recovering a Latin word from its Spanish, Italian, French, Portuguese and Romanian descendants, with reported accuracies above 70%. These models are trained on "cognate sets": groups of words assumed to descend from a common ancestor through regular sound change. Real cognate sets also contain borrowings. Some are words one Romance language took from another. Others are learned loans that scholars and the church re-imported from written Latin centuries later. A learned loan never went through the sound changes, so it looks almost like its Latin source. For a reconstruction model, that makes it an easy item to get right by copying, not a hard one.

This note argues that borrowing contamination does more than add noise. It may inflate benchmark accuracy and teach models to copy spelling instead of reversing sound laws. If so, the "error paradox" reported on the largest Romance benchmark, where the noisy full dataset scores higher than a clean subset, may be partly an artifact. We propose two things. The first is an audit that splits reconstruction accuracy by whether each word was inherited or borrowed. The second is a borrowing detector that needs no borrowing labels. It flags words that break the regular sound correspondences the model has learned, and it can be used to downweight those words in training. Romance is the test bed because it has expert borrowing labels. The payoff is in language families that have no such labels.

Core Claim

Our hypothesis has two parts, and each can be refuted. The first is that on ProtoRom, accuracy on borrowed and learned items is much higher than on securely inherited items, and that this gap accounts for a meaningful share of the advantage of the full dataset over the clean five-language subset. The second is that the size of a reflex's residual identifies borrowings well enough to be useful, and that downweighting the flagged items improves reconstruction measured on inherited words only. Here a reflex is one daughter-language word in a cognate set, and its residual is how badly the model predicts it from its sisters.

The detector matters most where labels do not exist, because the supervised alternative already works very well in Romance. What is new is not another pairwise classifier. It is treating "irregular with respect to the learned sound laws" as the operational definition of a borrowing, and evaluating reconstruction on the stratum that tests sound-law inversion, which is the inherited vocabulary.

Where This Sits

Three Romance papers from one group define the setting. RoBoCoP supplies the resource: 125,598 words with cognate and borrowing annotations across five languages. It also reports that performance consistently drops for Romanian, which it attributes to Slavic contact. Pater Incertus? shows that supervised classifiers separate cognates from borrowings almost perfectly when labeled pairs exist. Verba volant, scripta volant? builds the ProtoRom reconstruction benchmark from RoBoCoP. It names borrowing contamination as an unresolved source of error, but its models consume raw strings and never use the borrowing labels.

Our proposal reuses the first paper's labels as ground truth and treats the second paper's classifier as an upper bound. It motivates itself from the third paper's error paradox. That paper offers a competing explanation: more data lets the model learn regularities despite noise. We address this directly. The stratified audit is designed to tell the two explanations apart.

Cognate Transformer adapts a biological alignment model, which attends both along each word and across the aligned words of a cognate set, to reconstruction and to predicting a missing reflex. It assumes cognate sets are pure. We use its reflex-prediction setting as the engine that produces residuals. The PCA sanity check paper shows that horizontal transmission leaves a visible signature, which it calls "jogging" trajectories in a 2D projection of inferred trees. It is only a diagnostic after inference. Our detector instead acts at the level of individual words, and before training.

The non-linear mapping literature contributes one candidate ingredient, which we test rather than assume. The ingredient is a kernel-based outlier detector over correspondence features. Prior work used kernel PCA to make outliers linearly separable, in market manipulation detection. Kernel metric learning beat linear methods on curved class structure, reaching separability 4.74 against 1.25 for linear RCA on XOR in scalable kernel metric learning.

We considered and rejected the most obvious intersection of these directions. That idea is to scrub geographic confounding with Fair Kernel Decomposition, which extends linear null-space projection (INLP) to kernel spaces. We return to this below.

Research Program

The first component is the audit. Every ProtoRom test item is labeled as inherited, inter-Romance borrowing, or learned/unknown, using RoBoCoP's etymological annotations. Where RoBoCoP does not separate learned Latin loans from inherited words, we hand-annotate a stratified sample of roughly a thousand Latin-etymon items from etymological dictionaries. We do not know that the file's annotations cover the learned stratum, so this step may need to be our own work. We then re-score the published model types on each stratum: the character Transformer, CRF with reranking, the LSTM, and Flan-T5. We repeat this for the full dataset and for ProtoRom-all5.

The second component is the residual detector. For each cognate set, we hold out one reflex at a time and predict it from the remaining reflexes, plus the protoform when it is known in training. Prediction uses a model trained on cross-validation folds that exclude that set. Per reflex, we record three things:

  1. the edit distance between the predicted and actual reflex;
  2. the per-position surprisal along the alignment, meaning how unexpected each sound is to the model;
  3. a "conservatism ratio": distance to the protoform divided by mean distance to sisters, which is the spelling-history fingerprint described by Ciobanu and Dinu.

These features feed three detector variants of increasing complexity:

  1. thresholding the raw residual;
  2. a linear one-class or logistic model trained on synthetic injections only;
  3. the same model with an RBF kernel, approximated with Nyström landmarks so it scales to tens of thousands of sets.

The third component is repair. The detector's scores downweight training sets when the reconstruction model is retrained. This is a soft weight and does not drop sets, so the volume benefit from the error paradox is preserved. Detection and retraining can alternate for two or three rounds.

The fourth component is transfer. On families without borrowing labels, we cannot score detection on real borrowings. So we inject synthetic borrowings: a reflex replaced by a sister-language reflex or by the protoform itself, copied without sound change. On those families we measure detection recall and whether reconstruction recovers.

Baselines

The naive reference point is the published reconstruction models trained on ProtoRom as is, with no borrowing handling. We report their standard accuracy alongside our stratified accuracy. The strongest reported number is 73.1% raw accuracy for the character Transformer on the full ProtoRom dataset. On ProtoRom-all5, CRF-rerank reports 60.4% raw and 90.7% after linguists reviewed its errors; both are from Verba volant, scripta volant?. We retrain the Transformer with the same splits, hyperparameter budget and compute for every condition. Beating this baseline means a statistically significant gain in exact-match accuracy and edit distance on the inherited-only test stratum, without a loss on the full test set larger than the share of borrowed items.

The strongest existing competitor for the filtering step is the supervised cognate-versus-borrowing ensemble and character Transformer of Pater Incertus?. It reports 99.2% F1 on Italian-Romanian binary discrimination and 87.78% average F1 on the ternary Spanish-Romanian task. We train it on RoBoCoP pairs disjoint from the ProtoRom test sets, then use its probabilities as training weights exactly as our detector's scores are used. In Romance it is expected to win at detection. Our detector "beats" it only in the sense that matters for the claim: it recovers most of the supervised filter's reconstruction gain on inherited items while using no labels. We set "most" in advance as at least 70%.

A label-free heuristic baseline is the conservatism ratio alone, meaning distance to the Latin etymon. It is the cheapest detector of learned loans. Its known weakness is that genuinely conservative inherited words (Italian especially) also score as close.

Two ablations isolate our ingredients. The first compares linear and RBF-kernel detectors on identical features, which isolates the non-linear mapping. The second compares the residual features with and without the reconstruction model's surprisal, which isolates the regularity signal.

For non-Romance families, the reference points are the Cognate Transformer results: reflex-prediction edit distance 1.2786 on SIGTYP 2022 at 0.5 test proportion, against 1.4170 for the CNN baseline, and B-Cubed F1 0.7132 on the List et al. reconstruction benchmark. We rerun these models with and without synthetic injection and with and without our weighting.

Why not just…?

Why not just use the supervised borrowing classifier as a filter? In Romance this is a real threat. Given its reported F1, it may match or beat our detector there, and we report that outcome plainly. It needs labeled borrowings per language pair, and the file shows such labels only for the five Romance languages. Its reported weakness is also exactly where contamination matters most: Romanian, where RoBoCoP finds performance consistently drops. The transfer experiments and a Romanian-only breakdown settle whether a label-free signal adds anything outside the supervised regime. If it adds nothing even there, the contribution reduces to the audit.

Why not just add more data, since the Transformer already absorbs noise? The error paradox is the evidence for this view. That evidence, however, is accuracy measured on a test set that is itself contaminated. If learned loans are easy copy cases, more contaminated data would raise reported accuracy without improving sound-law inversion. The stratified audit tests this directly. If inherited-only accuracy also rises with volume, at the same rate as the full set, then the "more data" explanation wins and our repair step becomes unnecessary.

Why not just scrub geographic contact with kernel null-space projection? FKD removes a continuous confounder from a representation. The dominant contaminant in Romance is the learned Latin stratum, and it is not geographic: it entered every language from the same written source. Geography is closer to the right confounder for Romanian Slavic loans. Even there, projecting out location would also remove genuine regional sound changes, which is the signal we need. We therefore expect FKD on coordinates to hurt inherited-only accuracy. We include it as a single control on the Romanian subset rather than as a component.

Why not just use edit distance to the Latin etymon? This is the strongest cheap alternative for training-time filtering, because the protoform is known during training. It is run as a baseline. The ablation shows whether model surprisal adds anything beyond this distance on conservative inherited vocabulary. If it adds nothing, the simpler heuristic is what we recommend.

Decisive Checks

The audit is decisive on its own. If accuracy on the inherited stratum is within a few points of accuracy on the borrowed stratum, the shortcut hypothesis fails. In that case the error paradox stands as the original authors explained it, and we stop before building repairs.

If the gap is large, the detector is judged by two measures. The first is its ROC AUC against RoBoCoP borrowing labels, compared with the supervised ceiling. The second is the inherited-only reconstruction gain under soft weighting. The kernel component stays only if the RBF detector beats the linear one on held-out AUC under an equal tuning budget.

Transfer is judged by detection recall on synthetic injections in the List et al. and SIGTYP families, together with recovery of the clean-data edit distance. We will also release the stratified ProtoRom test split. Even if every modeling claim fails, that split makes future reconstruction numbers interpretable.