Which Correct Answer? Detecting Leaked Annotations by Surface-Form Preference

Many questions in extraction benchmarks have several right answers. Two annotators summarizing the same abstract will pick different but equally valid keyphrases. Two annotators marking who did what in a news event will draw the span boundary differently, one writing "South Korean vehicles" and the other "vehicles." A model that has never seen the benchmark has no reason to prefer the wording the official annotator happened to choose. A model that read the benchmark's answer key during pretraining does have a reason. We propose to detect this kind of leak, which the field calls benchmark data contamination, by measuring exactly that preference. Among the model's semantically correct answers, we ask how often it reproduces the published reference word for word compared with an equally valid alternative reference that was never released. A clean model should split roughly evenly once its general style is accounted for. A model trained on the answer key should tilt toward the published one.

The idea turns a known weakness of evaluation into a measurement tool. Exact string matching is a poor judge of quality, because it gives zero credit to valid paraphrases. But the distance between exact-match credit and semantic credit is not pure noise. Part of that distance is a fingerprint of memorized labels. The same reasoning shows that current contamination detectors, which compare scores on original and rephrased benchmarks under exact match, mix up two effects: memorization being removed, and the metric failing on reworded outputs. We will separate the two.

Core Claim

For a model $m$ and a set of items with two independent valid references, let $S$ be the outputs judged semantically correct against either reference. Let $p_{pub}$ be the fraction of $S$ that exactly matches the published reference, and let $p_{dec}$ be the fraction that exactly matches the private "decoy" reference. We define the surface-preference score as

$$\Pi(m) = \log\frac{p_{pub}(m)}{p_{dec}(m)} - \overline{\log\frac{p_{pub}}{p_{dec}}}_{\text{clean}},$$

where the second term averages the same log ratio over reference models known to be clean, such as models evaluated on post-cutoff items or models of the same family with no injected exposure. The correction removes ordinary stylistic alignment, for example the case where the published annotator just writes the way language models write.

We claim three things. First, $\Pi$ rises with label-level exposure, meaning training on test inputs paired with gold annotations, which is Level 4 in the taxonomy of DCR. Second, $\Pi$ stays near zero under input-only exposure (Level 3), because seeing the documents without the annotations gives no information about the annotator's choices. Third, $\Pi$ keeps working when the test inputs are paraphrased, which is the regime where The Emperor's New Clothes in Benchmarking? found that rephrasing-based defenses offer near-zero resistance. The second prediction can falsify the method. If $\Pi$ moves under input-only exposure, then it measures familiarity with the documents and not leaked labels.

Why This Intersection, and What Is Already Done

Three lines of work motivate the proposal by showing that exact match badly misjudges open-form outputs. BEMEAE replaces exact span matching in event argument extraction with a deterministic normalization pipeline. It reports that exact match correlates with human rankings on RAMS at Kendall's $\tau$ of only 0.44, and that BART-Gen moves from 7th to 2nd place under the new scoring. RAEE uses an LLM judge and finds precision rises 20.52% on average across 14 models. KPEval trains a phrase-level paraphrase matcher for keyphrases, called SemF1, that tracks human judgment better than BERTScore does. All three treat the gap between exact and semantic credit as error to remove. We instead treat part of it as a signal.

The contamination work supplies the detectors that we stress-test. ConStat flags contamination as an unexplained drop from an original benchmark to a rephrased reference benchmark. SSA swaps named entities and watches for flipped predictions. CLEAN-EVAL paraphrases and back-translates test items. All three score outputs against a single reference with standard accuracy or string metrics. SSA works only on a 6-class labeling task, so it never meets the open-vocabulary problem at all.

The constraining result is the fidelity-versus-resistance trade-off reported in The Emperor's New Clothes. Light rephrasing keeps the task intact but does not stop contaminated models. Aggressive rephrasing stops them but changes the task, with fidelity falling to 0.686 on MMLU. Our detector avoids this trade-off because it never edits the test item. It changes only the reference that outputs are compared against.

Empirical Study of Zero-shot Keyphrase Extraction with Large Language Models supplies the concrete open problem. It names both exact-match bias and likely contamination of Inspec, SemEval, and DUC2001 as unaddressed threats to its own Llama3 and Gemma2 results, and it does not separate them. NLP Evaluation in trouble shows that annotation contamination is real: chat models reproduce CoNLL-2003 labels with perfect accuracy. However, its verbatim-recitation test needs the model to regurgitate on request.

The closest existing idea is to rerun ConStat with a semantic scorer. That would fix the confound but would still detect contamination only through score drops on rephrased inputs. Our method never rephrases inputs, and its signal is the choice among correct answers, not the level of performance.

Research Program

The program has three parts.

The first part builds decoy references. For a subset of items from Inspec, SemEval2017, KPTimes, and KP20k (keyphrases), and from RAMS and WikiEvents (event arguments), we commission fresh annotations under the original guidelines. The annotators never see the published gold labels. We keep these annotations private. For event arguments, the decoy keeps the role assignments and lets the annotator choose span boundaries and mention choices freely. This targets precisely the boundary idiosyncrasies BEMEAE documents. A few hundred documents per dataset should be enough because the statistic is paired within items. We will also report how often the two annotators agree on surface form, since the method needs room to disagree.

The second part builds a symmetric semantic judge. Correctness is judged against the union of the two references using deterministic matchers: BEMEAE's rule pipeline for event arguments and SemF1 for keyphrases. This choice avoids a judge that may itself have memorized the published labels. As a sanity check, we will confirm that the judge accepts published-matching and decoy-matching strings at equal rates on outputs from clean models. An LLM judge in the style of RAEE serves only as a secondary check.

The third part injects contamination under control. We continue pretraining open models of 7–8B parameters, Llama-3-8B and Qwen2.5-7B, which fits the scale that A Narrowing Geometry in Contaminated Reasoning found tractable. Each run exposes a random half of the test items at one of three levels. The first is label-level: inputs with published gold labels. The second is input-only. The third is semantic-level, where the label content is reworded, following DCR's Level 1. The other half of the items stays unexposed, so each run gives within-model ground truth. We also apply the decomposition to frontier open models where contamination is suspected, but we treat those results as descriptive.

Baselines

The naive reference point is the exact-match drop used without any semantic correction. It compares a model's exact-match score on exposed versus unexposed items, or on original versus rephrased items. This is what Kang & Shin implicitly rely on. We report its detection area under the ROC curve on the injected splits. We expect it to be confounded, because rephrased items lower exact-match scores even for clean models.

The strongest existing method is ConStat. Its reported result on Lacuna is a 98.4% detection rate for controlled contamination, against 71% for overlap-based baselines, on its own task suite, which includes ARC-Challenge. It also reports, for example, an 8.3% contamination gap for Mistral-7b-v0.1 on GSM8K. We found no reported result on Lacuna for ConStat on keyphrase or event-argument benchmarks. We will run ConStat in two versions: as published, scoring the rephrased reference benchmarks with exact match, and as a semantically scored variant using the same symmetric judge we use. Both get the same models, the same injected runs, the same reference-model pool, and the same rephrasing budget. ConStat with semantic scoring is the real competitor. Our method counts as beating it if it reaches higher detection AUC on label-level injection when inputs are also rephrased, and if it matches ConStat elsewhere at lower cost, since it needs no difficulty-curve fitting over many reference models.

We also run the verbatim continuation probe from NLP Evaluation in trouble and DCR's multi-level prompt probes, with the same models and prompt budgets.

Two ablations isolate the key ingredient. The first drops the decoy and uses raw $p_{pub}$ among correct outputs, corrected only by clean models. This tests whether a second human reference is needed or whether style correction is enough. The second replaces human decoys with decoys written by an LLM. This tests the cheaper path and exposes the bias of comparing against text written in a model's own style.

Why not just…?

Why not just grade everything with an LLM judge and rerun existing detectors? A judge in the style of RAEE removes the exact-match confound and agrees well with humans (93.49% recall agreement for DeepSeek-r1). We include this as the semantic ConStat baseline. But it answers a different question: whether scores drop when inputs change. Emperor's New Clothes shows that contaminated models map rephrased inputs back to what they memorized, so a drop may never appear. A judge from a frontier model may also be contaminated with the same published labels and favor them. The symmetric-acceptance check measures this directly. If semantic ConStat matches our AUC under rephrasing, our method's value shrinks to cost and to its specificity for labels, and we will say so.

Why not just paraphrase the benchmark and keep exact match? That is CLEAN-EVAL's approach. Emperor's New Clothes reports near-zero resistance for surface rephrasing. Our part (A) also predicts that much of any drop CLEAN-EVAL sees on open-form tasks is caused by the metric. We test this directly: for clean models, the drop from paraphrase under exact match minus the drop under semantic scoring estimates the share due to the metric.

Why not just search the pretraining corpus for n-grams or ask the model to recite the test set? Corpora for most models are closed. ConStat reports that overlap-based detection catches 71% of controlled cases versus 98.4% for its own method. Recitation probes also test whether the model remembers the inputs, which our input-only condition treats as a different phenomenon. Our statistic asks whether the model remembers the annotator's choices, which recitation detects only when the model is willing to regurgitate.

Proposed Experiments

The first experiment decomposes existing detector gaps. On the injected runs, we compute ConStat's gap under exact-match scoring and under symmetric semantic scoring, on RAMS, WikiEvents, KPTimes, and TempAnswerQA from Time to Revist Exact Match. On TempAnswerQA we additionally score with sMAPE, to see whether contaminated models hit dates exactly while clean models miss by ±1. The decisive quantity is the share of the exact-match gap that remains on unexposed items for clean models. Any such remainder is an artifact of the metric.

The second experiment is the main detection test. On each injected run, we compute $\Pi$ on exposed versus unexposed halves and report AUC by exposure level, with and without paraphrased inputs. The method succeeds if $\Pi$ separates label-level exposure with AUC above semantic ConStat under rephrased inputs. It must also stay near zero under input-only exposure. Movement at the input-only level would refute the claim that $\Pi$ is specific to labels. We also measure a dose-response curve over the number of exposure epochs.

The third experiment is an audit. We apply $\Pi$ to the Llama3 and Gemma2 zero-shot keyphrase setting from Kang & Shin on Inspec, SemEval2017, and KPTimes, and to RAMS event-argument extraction with publicly available models. This gives the first estimate of how much of their exact-match and SemF1 scores reflects leaked labels rather than extraction ability. We report confidence intervals from bootstrapping over items, and we flag results as inconclusive wherever the two annotators agreed on surface form too often to leave room for a preference.