Annotation Quirks as Contamination Canaries in Universal Dependencies

Multilingual language models are routinely scored on Universal Dependencies (UD), a collection of treebanks for over a hundred languages that label every word with a part of speech, morphological features, and a syntactic relation to another word. These treebanks are public text files, and they have been on the open web for years. A model fine-tuned on a treebank's training split may therefore score well on its test split partly because it read that test split during pretraining. Output-level contamination tests can flag that something leaked. They cannot say how many points of the reported score the leak is worth, and the stronger tests need a clean reference model, which does not exist for modern multilingual models.

This note proposes using the treebanks' own inconsistencies as the measuring instrument. UD is full of labeling choices that annotators made inconsistently: the same word in the same context receives one label in the training split and a different one in the test split. A model that learned from the training data cannot predict these test-only choices; at best it follows the training convention. A model that memorized the test file can reproduce them. The rate at which a fine-tuned model "correctly" reproduces unpredictable test-split quirks, compared with the same rate on matched sentences the model could not have seen, gives a black-box, reference-free, graded estimate of how much memorization inflates its score. In effect, the quirks act as accidental canaries planted by human annotators.

Core Claim

Cross-linguistic annotation inconsistency has so far been treated as noise that blurs evaluation. Probing work finds that it skews comparisons across languages (Morph Call), and pooling corpora with different conventions hurts Hungarian taggers (Variety delights (sometimes)). We claim the same inconsistency is also a signal. Take every test token whose gold label contradicts the training-split majority label for the same word in the same context. For each such token, the label a model trained only on the training split should predict is known in advance. Above-baseline agreement with the contradicting gold label can only come from information about the test file itself. The excess yields a lower bound on score inflation per treebank and per language, using nothing but model predictions.

Let $I$ be the set of test tokens whose gold label contradicts the training-split majority label for their context, and $N$ the number of test tokens. Let $H_{\text{test}}$ be the fraction of $I$ on which the fine-tuned model outputs the gold label. Let $H_{\text{fresh}}$ be the same fraction on matched contradicting tokens drawn from sentences the model cannot have seen. Matching is on language, relation or tag type, and how often the context occurs in training. The estimated inflation is

$$\hat{\Delta} = \big(H_{\text{test}} - H_{\text{fresh}}\big)\cdot\frac{|I|}{N},$$

which counts only the memorized points that happen to fall on quirky tokens. It is therefore a floor on total inflation, not an estimate of it. Memorized "easy" tokens are invisible to this measure.

Research Program

The first component is finding the canaries. We adapt the variation-nuclei method of Assessing the Annotation Consistency of the Universal Dependencies Corpora, which finds identical strings in identical contexts that carry different labels. That paper reports that lemma-based matching reaches 62% precision for English and 65% for French but only 19% for Finnish, rising to 72% when surface wordforms replace lemmas. The paper therefore constrains our design: we match on wordforms for morphologically rich languages and on lemmas elsewhere. We do not need every flagged pair to be a true annotation error. We only need the training split to predict a label different from the test gold, and we check that directly. We add phenomenon-level canaries documented in the literature: the expl relation, used by only 27 of 60 languages in UD v2.1 at frequencies spanning more than an order of magnitude (Expletives in Universal Dependency Treebanks), and orthography-driven compound splitting (Compounds in Universal Dependencies).

The second component is a cross-treebank natural experiment. Several languages have multiple UD treebanks with documented convention differences. The three Hungarian corpora diverge on auxiliaries, proper nouns, NumType, and PronType (Variety delights (sometimes)). We fine-tune on treebank A and evaluate on treebank B's test split. A model that learned only from A should apply A's conventions wherever the two diverge. Systematic drift toward B's conventions on exactly B's test sentences, but not on fresh sentences annotated under B's guidelines, indicates memorization of B.

The third component is the fine-tuning regime itself. We use full-parameter fine-tuning, meaning every weight is updated, as the main regime. A Closer Look at Few-Shot Crosslingual Transfer found it beats frozen-layer and cosine-classifier alternatives across 40 languages, peaks after about one epoch, and relies on shallow lexical cues. That combination motivates our open question. How Fine-Tuning Allows for Effective Meta-Learning argues that full fine-tuning corrects misaligned representations globally. If so, it may overwrite memorized test annotations with training conventions. Alternatively, by touching every layer, it may re-surface them. We measure $\hat{\Delta}$ as a function of fine-tuning epochs and compare against low-rank adaptation (LoRA, which trains small add-on matrices while freezing the backbone). CANARY found that low-rank updates partly hide contamination from hidden-state detectors, with AUROC around 0.83 at rank 4. Our output-level measure tests whether they also change how much memorized content reaches predictions.

The fourth component is calibration with ground truth. We take an open model, run continued pretraining on a general corpus, and inject selected UD test files at controlled doses of zero, one, and ten exposures. Uninjected treebanks serve as controls. Because the true contamination status is known, we can check that $\hat{\Delta}$ is near zero at dose zero, grows with dose, and tracks the true score gap between injected and uninjected runs.

Baselines

The naive reference point is to report fine-tuned accuracy and labeled attachment score as is, assuming no contamination. This is current practice, for example in Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models, which treats UD POS labels as ground truth while correlating transfer with leaked token volume ($\rho$ up to 0.67). Any audit we propose must at least change conclusions relative to this number on some languages.

The strongest existing black-box competitor is the exchangeability test of Contamination Report for Multilingual Benchmarks. It compares a model's likelihood of a benchmark in canonical order with its likelihood under shuffled orders. That paper flags Llama-3.1-8B and Gemma-2-9B on nearly every benchmark at $p < 0.0013$ and finds Aya-23-8B clean on XCOPA and PAWS-X. It does not report results on UD, so no reported result was found on Lacuna for exchangeability testing of UD treebanks. We run it ourselves on raw sentence sequences of each test file, with the same models and the same GPU budget. Our method beats it if, on the calibration models, it detects the one-exposure dose at a rate at least matching exchangeability while also recovering the true score gap, which exchangeability cannot provide. We also expect it to cost far less than the 362 hours on 8 H100s the exchangeability report needed, because it needs one fine-tuning run and one decoding pass.

For the fine-tuning backbone itself, the reference is standard full fine-tuning as in A Closer Look at Few-Shot Crosslingual Transfer. The file contains no numeric UD POS accuracy for it, so no reported result was found on Lacuna for UD POS accuracy of a fully fine-tuned Llama-3.1-8B. We reproduce it with matched hyperparameter search across all conditions.

Where pretraining data is public, as for the calibration model, we include n-gram overlap between test files and the pretraining corpus as an oracle baseline.

The key ablation replaces the canary set $I$ with test tokens whose gold label agrees with the training majority, matched on frequency and tag type. If excess accuracy appears equally there, the signal reflects general difficulty or domain effects, not memorization of test-specific choices. A second ablation uses a model trained from scratch on the training split only, with no pretraining. It estimates how often the contradicting gold label is recoverable from training data alone. This guards against flagged "quirks" that are actually predictable from wider context.

Why not just…?

Why not just run the exchangeability test on the treebank files? It is a real competitor for detection, and we run it. It answers a different question: whether a file's ordering was seen. It gives a $p$-value, not points of inflation, and the contamination report itself notes it cannot quantify score inflation. A fine-tuned tagger's score could also be inflated by memorized label strings even when sentence order carries little signal. The calibration experiment decides which test is more sensitive at low dose. If exchangeability dominates on detection, our contribution narrows to quantification, which is still the missing piece.

Why not just search the pretraining data for the test files? For most audited models the pretraining data is not available. Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models shows that even nominally filtered English corpora carry hundreds of millions of non-English tokens. Presence of a string also does not tell you whether it affected the score. We use overlap search where data is open, as a check on our estimate, not as a replacement.

Why not just evaluate on fresh treebanks released after the model's cutoff? This is the most serious alternative. Where fresh data exists, it is the cleanest test, and we use new sentences added to existing treebanks in post-cutoff releases as our $H_{\text{fresh}}$ control. Fresh data alone, though, cannot correct the large body of published numbers on existing test splits. It also does not exist for many low-resource languages, and new treebanks differ in domain and annotators, which confounds score comparisons. Our measure applies to the existing splits. Fresh data only needs to supply a matched quirk rate, which takes far fewer sentences than a full replacement benchmark.

Why not just use hidden-state auditing such as CANARY? CANARY detects 1% fine-tuning contamination with AUROC 1.000 by comparing hidden states against a trusted base model. Here, the contamination we care about sits in the base model itself, and the multilingual exchangeability results suggest no trusted multilingual base is available. Our method needs no reference model and works on prediction-only APIs. As an optional secondary analysis on Llama-3.1-8B, we check whether canary tokens activate features from the Llama Scope sparse autoencoders differently from matched tokens. This analysis is supporting evidence, not the method.

Proposed Experiments

We audit Llama-3.1-8B, Gemma-2-9B, Qwen2.5-7B, and Aya-23-8B. The choice deliberately spans models judged more and less contaminated by the exchangeability report. Each is fully fine-tuned, and separately LoRA-tuned, on UD POS tagging, morphological features, and dependency relations. The treebanks cover the languages with the strongest documented inconsistencies: English, French, Finnish, Dutch, German, Czech, and the three Hungarian treebanks. We report $\hat{\Delta}$ with bootstrap intervals per treebank and per model, alongside raw scores.

The decisive checks are four.

The first is on the calibration model. $\hat{\Delta}$ must be indistinguishable from zero on uninjected treebanks, must rise monotonically with injection dose, and must correlate with the true injected-versus-uninjected score gap.

The second is the matched-agreeing-token ablation. Excess accuracy must be concentrated on canary tokens.

The third concerns the Hungarian cross-treebank design. Convention drift toward the evaluated treebank must appear on its test sentences but not on fresh sentences annotated under the same guidelines.

The fourth concerns the audited models. If $\hat{\Delta}$ ranks Aya-23-8B below Llama-3.1-8B, it agrees with the exchangeability report. If its per-language values change which languages appear "easy," it shows that existing UD-based conclusions about cross-lingual ability, including the leaked-volume correlations, are partly measurements of memorized annotation.

A null result on the calibration model would be informative too. It would mean single-exposure memorization of treebank labels does not survive fine-tuning to reach predictions, which bounds how much contamination can matter for UD scores.