Contested Labels as Contamination Canaries

When annotators genuinely disagree about a benchmark item, the official "gold" label is partly an accident. It reflects which three or five people happened to label the item and how their votes broke. If many more people label the same item, the majority often lands somewhere else. A model that learned the task honestly has no way to recover that accident. A model that saw the test set with its answers during training can recover it. This proposal turns that asymmetry into a contamination test. On items where the released gold label departs from a dense crowd's view, we measure how often a model agrees with the gold label beyond what its general skill predicts. Excess agreement there is evidence that the model memorized answers, and the test needs neither paraphrasing nor access to training data.

This matters because current contamination detectors are in poor shape. Benchmark contamination means test items leaking into a model's training data. A controlled study found that leading detectors barely correlate with one another. The same study found that they go blind when leaked answers are rewritten with chain-of-thought explanations during instruction tuning (Towards Data Contamination Detection for Modern Large Language Models). Detectors that rely on rephrasing confuse memorization with the extra difficulty that rephrasing introduces (Confounders in Instance Variation). They also confuse it with a model's simple preference for fluent text. Every existing method also assumes a single correct answer. Work on disagreement categorization, which sorts why annotators diverge, has meanwhile produced exactly the item-level structure that could separate "the model knows the task" from "the model knows this answer key."

Core Claim

On a contested item, the released gold label disagrees with, or barely wins over, the label distribution from many annotators. On such items, a model's agreement with the gold label has two sources. One is task competence, which should mostly push the model toward the crowd's view. The other is exposure to the answer key, which pushes it toward the gold label specifically. We first calibrate, across a fleet of reference models, how gold agreement on contested items normally tracks accuracy on uncontested items. Any model sitting far above that curve is flagged for label-level contamination.

Write $C$ for the contested items and $U$ for the uncontested items, where gold equals a clear crowd plurality. Write $A^{U}(m)$ for model $m$'s accuracy on $U$ and $A^{C}_{gold}(m)$ for its agreement with the released gold label on $C$. The contamination score is

$$\Delta_m = A^{C}_{gold}(m) - H\big(A^{U}(m)\big),$$

where $H$ is a monotone curve fit on reference models believed clean. The construction borrows ConStat's idea of a hardness-correction curve fit across reference models (ConStat). We apply it along a different axis: the reference is the same items judged by a dense crowd, not a rephrased or synthetic benchmark. Because nothing is rephrased, the masking effect and the fluency trap cannot arise.

The claim has a sharper, falsifiable corollary. The signal should separate data-level contamination, where only questions leaked, from label-level contamination, where questions leaked with answers. These are two of the four contamination levels defined in Benchmark Data Contamination of Large Language Models: A Survey. Question-only exposure should leave $\Delta_m$ near zero, because the accident lives in the label. As far as we can tell from the literature, no current detector separates these two levels.

Where Disagreement Categorization Comes In

Not all contested items are equally useful canaries, and this is where the disagreement literature does real work. From Dissonance to Insights separates data-centric disagreement (missing context, genuine textual ambiguity) from annotator-centric disagreement (lapses, noise, individual subjectivity). Iterative development of family history annotation guidelines makes a parallel split into semantic errors, caused by gaps in domain knowledge, and pragmatic errors, caused by unclear instructions. Mapped onto our setting, the categories make different predictions.

Where the gold label is an annotator-centric error, a competent clean model should actively disagree with gold, so contamination produces the largest gap. Where the item is genuinely ambiguous, gold is close to a coin flip, and a clean model's agreement should sit near the crowd's probability for that label. Where the disagreement is pragmatic, meaning it comes from guideline conventions, a clean model fine-tuned on the training split can legitimately learn the convention and match gold. These items are the main source of false positives and must be identified and either down-weighted or modeled separately.

So the taxonomy is the instrument that decides which contested items carry signal. Two further sources motivate the stratification. Insights into developing analytical categorization schemes shows that negated, implicit, or sparse phrasing systematically lowers human agreement. NUTMEG shows that spam and incompetence can be statistically separated from valid perspectives; it reports 0.81 correlation with true annotator competence against 0.58 for MACE on synthetic data. NUTMEG, however, needs annotator demographic metadata that most NLI corpora lack. We therefore use it only on POPQUORN, where that metadata exists. Elsewhere we rely on human coding of a sample under the data-centric versus annotator-centric scheme, plus an automatic proxy trained on that sample.

Research Program

The first component is building canary sets. ChaosNLI collects roughly 100 labels per item on items drawn from existing NLI benchmarks with released gold labels. It is the primary testbed, because the gap between a five-way vote and a hundred-way vote is directly measurable. CommitmentBank, with at least eight Likert annotations per item (Identifying inherent disagreement in natural language inference), gives a second NLI-style set. HateXplain and the other hate speech corpora used by Crowd-Calibrator extend the test to subjective classification. MultiPref, with four annotators per pair, and HelpSteer2, with three to five, extend it to preference data. For those preference sets, the "chosen" response in a split vote plays the role of gold. Diverging Preferences found more than 30% of pairs divergent there. That paper treated divisive items as noise to prune from evaluation; we treat them as the most informative items for auditing. For each set we define $C$ and $U$, code a stratified sample of $C$ by disagreement category, and train the category proxy.

The second component is calibration. A reference fleet consists of open models whose pre-training predates the benchmarks or whose data has been inspected. To capture the legitimate convention-learning effect, the fleet also includes the same models fine-tuned only on training splits. The curve $H$ is fit per benchmark and per category stratum, and bootstrap confidence intervals are computed over items.

The third component is a controlled oracle. We fine-tune Llama-3.1-8B and Qwen2.5-7B on test sets under several conditions crossed with duplicate counts. Inputs only gives data-level contamination. Inputs with gold labels gives label-level contamination. Inputs with gold labels plus model-generated chain-of-thought rationales reproduces the answer-augmentation condition that defeats current detectors. A fourth condition uses inputs with crowd-majority labels; this is contamination with a "corrected" answer key, and it should make $\Delta_m$ negative on contested items. We vary duplicate counts because Data Contamination: From Memorization to Exploitation shows that memorization often goes unexploited until duplication is heavy. On SNLI with BERT, 200 duplicates raised exploitation to 40%. Only items whose answers are actually exploited should move the signal.

The fourth component is a secondary signal: the collapse of predictive entropy on contested items. Crowd-Calibrator measures the distance between a model's output distribution and the crowd distribution. We test whether label-level contamination collapses that distribution toward gold, rather than toward the majority. This is the open question in Gap 3 of the research file.

Baselines

The naive reference point is raw gold agreement on contested items, with no calibration curve and no category stratification. This baseline is the simplest version of the idea, and it is also the ablation that isolates our two key ingredients. A uniformly stronger model will score higher on it without any contamination. We will report its false-positive rate on the clean fine-tuned reference models; we expect it to be high, and we must show this rather than assume it. A second ablation keeps the calibration curve but pools all contested items, which tests whether the taxonomy adds anything beyond calibration.

The strongest existing behavioral detector is ConStat (ConStat), which reports detecting 89% of syntax-contaminated and 98% of sample-contaminated models. It also reports estimated inflation of about 6.6% for Llama-3-70b on ARC and about 8.2% for Mistral-7b on GSM8k. ConStat is the closest competitor because it also corrects for difficulty using a reference fleet. We will run it on the same oracle models, with the same reference fleet. Its reference benchmarks will be rephrased versions of the same NLI, hate speech, and preference test sets, and its tuning budget will match ours. Beating it means a higher detection rate at a matched false-positive rate on the answer-augmented condition and on subjective benchmarks. It also means correctly assigning a near-zero score to the data-level condition, which ConStat is not designed to separate.

PaCoST (PaCoST) tests whether a model is more confident on originals than on paraphrases, without training-data access. No reported detection rate for it was found on Lacuna; the file reports only qualitative flags across anonymized models and validation on WMDP. We will run PaCoST with its published significance threshold on identical items and identical models.

Min-K% Prob and token-overlap matching will also be run, as the methods Towards Data Contamination Detection for Modern Large Language Models reports failing under answer augmentation. Where training data is available, as in our oracle models, n-gram overlap serves as an upper-bound sanity check.

On the disagreement side, the Artificial Annotators ensemble (Identifying inherent disagreement in natural language inference) reports 65.60% accuracy and a 66.49 F1 on the disagreement class on CommitmentBank. It is not a contamination detector. We use it only as an alternative automatic way to identify contested items, compared against dense crowd labels.

Why not just…?

Why not just use n-gram overlap or Min-K% Prob? Frontier training corpora are private, so overlap cannot be computed for the models people most want to audit. Token overlap found almost no exact matches on modern models even where other evidence pointed to leakage. Under answer-augmented fine-tuning, Min-K% and canonical-order tests gave no clear signal even at full contamination (Towards Data Contamination Detection for Modern Large Language Models). Our oracle includes exactly that condition. If Min-K% detects it as well as $\Delta_m$ does, our claimed advantage disappears, and we will report that.

Why not just paraphrase items and look for a drop? Rephrasing changes difficulty: the masking effect in Confounders in Instance Variation. For subjective items it can also shift the item into a different ambiguity category, such as implicit or negated phrasing, which Toivanen et al. show lowers human agreement. Our test holds the item fixed and changes only the reference label, so the difficulty confound cannot arise by construction. The comparison with ConStat and PaCoST on subjective sets measures whether this matters in practice.

Why not just compare gold agreement to crowd-majority agreement without any taxonomy? This is the real threat. It is our pooled ablation. If the calibrated pooled score matches the stratified one, the honest conclusion is that disagreement categories are unnecessary for detection. The contribution would then shrink to "dense-crowd relabeling as a contamination reference." The experiment that settles it is the false-positive rate on clean models fine-tuned on training splits. Convention-learning should inflate gold agreement specifically on pragmatic, guideline-driven items. If it does not, the taxonomy is redundant.

Why not just drop divisive items, as Diverging Preferences suggests? Pruning makes evaluation cleaner, but it removes the items where answer-key exposure is most visible. A pruned benchmark leaves only uncontested items, and on those a clean model and a contaminated one both choose the gold label, so the two cannot be told apart.

Proposed Experiments

The decisive check is the oracle grid. Across conditions and duplicate counts, $\Delta_m$ should rise with label-level exposure and stay flat under input-only exposure. It should turn negative under majority-label exposure. It should remain detectable under chain-of-thought answer augmentation. Its rise should be steepest in the annotator-centric stratum. We report detection AUROC over oracle models and false-positive rate on the clean reference fleet. We also report rank correlation between $\Delta_m$ and the true duplicate count, plus agreement with the oracle's ground truth, so that the inconsistency crisis is measured rather than repeated.

A second check targets the confound. Models fine-tuned only on training splits must score $\Delta_m$ within the bootstrap interval of zero once pragmatic items are handled. If they do not, the method mistakes learning guidelines for leakage, and the claim fails.

A third check applies the calibrated detector to public open-weight and API models on ChaosNLI, CommitmentBank, the hate speech sets, and MultiPref. Model-level flags are compared with ConStat's and PaCoST's. Where available, we also compare against the narrowing-geometry signature from A Narrowing Geometry in Contaminated Reasoning. That paper explicitly left open whether its signature appears in multi-perspective domains, and our labeled oracle models provide a direct test of it.

A last measurement tracks the entropy of the model's output distribution on contested items under each oracle condition. This tests whether label-level contamination is one cause of the overconfident collapse that Crowd-Calibrator and Diverging Preferences observed in large language models on subjective data.