Label Inertia: Detecting Memorized Satisfaction Labels in LLM Usefulness Judges

Search engines and chat assistants are increasingly graded by large language models (LLMs) acting as judges. Given a user's search session, a judge estimates how useful each document was to that user, or how satisfied the user was with a reply. These judges are validated by measuring their agreement with human labels on a small number of public behavioral logs. Two examples are THUIR-KDD'19 and TianGong-QRef, which record what users queried, clicked and dwelled on, together with their satisfaction and usefulness ratings. Those logs have been public on the open web for years. Nobody has checked whether the judges being validated on them have already read them, labels included. If they have, the reported agreement partly measures recall, not judgment.

The usual contamination tools do not apply here. Temporal splits need data published after a model's training cutoff, and these logs predate every current model. Paraphrasing hides contamination rather than removing it. Satisfaction labels have no answer key to search for. We propose a detector built for subjective labels, called label inertia. We edit a session so that a careful human would change the label, for example by appending a follow-up turn in which the user rejects the result, or by placing an equivalent document earlier in the session so the target becomes redundant. A judge that is reasoning about the session should move its score with the humans. A judge that has memorized the recorded label should keep reproducing it. We calibrate this test with deliberate contamination of an open model, then apply it to the judges the field currently trusts.

Core Claim

Our claim is that, for subjective evaluation labels, contamination leaves a measurable trace. On items the model has seen together with their labels, the judge's score stops responding to edits that humans agree should change the label. Unseen items do not show this trace. We further claim that this responsiveness gap identifies exposed items better than likelihood-based membership probes do. Those probes only show that the text was seen. Responsiveness shows whether the label is driving the score.

Let $s(x)$ be the judge's usefulness score for item $x$. Let $x'$ be a label-changing edit of $x$, and let $\Delta h(x)$ be the human-validated shift in the label caused by the edit. We define responsiveness on an item set $D$ as

$$ R(D) = \frac{\sum_{x \in D} \left(s(x) - s(x')\right)}{\sum_{x \in D} \Delta h(x)}, $$

so $R=1$ means the judge moves as much as humans do and $R=0$ means it does not move at all. The contamination signal is $R(\text{fresh}) - R(\text{suspected})$ for the same judge and the same edit types.

Grounding in Prior Work

Two papers motivate the work by documenting the failure modes we suspect contamination is amplifying. LLM-Driven Usefulness Labeling for IR Evaluation prompts six LLMs to label document usefulness on THUIR-KDD'19 and TianGong-QRef. It reports Spearman correlations of 0.33–0.48 with humans, best for Llama 3.3 70B. It also finds a "relevance bias": judges treat any on-topic document as useful even when it is redundant. That paper runs no exposure probes, and we take its benchmark and judges as our starting point. Rethinking the Evaluation of Dialogue Systems shows on ReDial that human usefulness medians fall from 70 to 56 once annotators see the user's follow-up turn. Agreement with expert satisfaction rises from ρ=0.55 to 0.66 in the same condition, while GPT-3.5-turbo's scores do not move. That paper attributes the rigidity to the judge without asking why. We reuse its follow-up-turn manipulation as one of our edits.

Two papers constrain the design by ruling out the cheap detectors. Test of Time shows that temporal decay disappears once memorized arXiv content is rewritten into reasoning questions (−3.83 points on cloze items versus +2.19 on transformed physics items). In the same paper, influence-function tracing on OLMo2-7B-Instruct finds the source document 77.5% of the time for cloze items but only 17.5% for transformed ones. Detection that relies on surface form therefore fails exactly when contamination is laundered. Search-Time Data Contamination shows that public mirrors strip canary strings and let agents copy answer keys, raising Sonar Deep Research on HLE from 17.4% to 37.5%. It studies only factoid QA with deterministic keys. Our setting has no key, which is why we need a behavioral test. The contamination survey supplies the distinction between data-level exposure (questions only) and label-level exposure (questions with gold labels), which we manipulate directly. LiveBench and DyVal escape contamination by moving to objectively scored tasks. That route is closed for satisfaction, which is subjective by definition. We keep human labels and instead test whether the judge is reasoning.

The two dialogue-side datasets come from Simulating User Satisfaction, whose USS collection provides turn-level 5-point satisfaction labels across ReDial, MultiWOZ, SGD, JDDC and CCPE. These labels let us extend the test from search logs to conversations.

Research Program

The work has three parts.

Edit construction and validation. We build three families of edits. The first two are label-changing. Feedback edits append a user turn or a reformulated query that signals rejection, following the setup of Siro et al. Redundancy edits insert, earlier in the session, a clicked document that carries the same content as the target; this targets the relevance bias directly. The third family is a control. Neutral edits shuffle unrelated earlier queries or rephrase cosmetically, and they should not change the label.

Crowdworkers relabel a sample of edited items to estimate $\Delta h$ for each edit type and to confirm that neutral edits leave labels unchanged. Without this validation, low responsiveness could reflect edits that humans also ignore.

Calibration by controlled contamination. We continue-pretrain an open-weights model whose training data is documented (OLMo2-7B, the model used in Test of Time) on part of the THUIR-KDD'19 and USS data. We hold out matched sessions from the same tasks. We vary two factors: exposure level (sessions alone versus sessions with labels) and repetition count. Afterwards we instruction-tune identically, and we prompt the judge with the DNA template from Dewan et al.

We record four things on injected and held-out items: Spearman agreement with human labels, $R$ under each edit family, perplexity-based membership scores, and influence-function hit rates. This gives ground truth for whether label inertia tracks label-level exposure.

Audit of deployed judges. We apply the calibrated test to Llama 3.3 70B, GPT-4o-mini and GPT-3.5-turbo. These are the judges whose numbers the field currently cites. Each judge sees the original logs and a fresh comparison set built from the same sources: a few hundred newly recorded sessions on new search tasks, plus newly written dialogues. Both fresh sets are labeled with the original annotation protocol. We then report each judge's agreement on fresh versus public data alongside its responsiveness gap.

Baselines

The naive reference point is to take the published agreement at face value. That means Spearman ρ between 0.33 and 0.48 on THUIR-KDD'19 and TianGong-QRef for the best judges, as reported in LLM-Driven Usefulness Labeling for IR Evaluation, with no exposure check. We rerun that protocol with the same prompts and models to reproduce this range before anything else. Any later claim of inflation is measured against our reproduction, not the paper's numbers.

The strongest existing detector is likelihood-based membership inference. We compare the judge's perplexity on the logged session text against reference text of matched length and domain, a matching-based method catalogued in the contamination survey. In the calibration model we also run influence-function tracing as in Test of Time. That paper reports a 77.5% top-1 hit rate on verbatim items, but no reported result was found on Lacuna for either membership inference or influence tracing applied to behavioral search logs or satisfaction labels. Both detectors and label inertia receive the same items, the same model access and the same number of forward passes per item. Any thresholds are tuned on a shared development split. Label inertia beats these detectors if, in the controlled-contamination model, it separates label-level injected items from held-out items with higher AUC. It should also separate label-level from data-level exposure, which perplexity cannot do because the text is identical in both conditions.

Two ablations isolate the key ingredient. The neutral-edit control checks that a low $R$ reflects label anchoring and not general insensitivity to edits. The data-level-only injection condition tests whether exposure to session text alone, without labels, produces inertia. If it does, the mechanism is something other than label memorization.

Why not just…?

Why not just run a membership-inference or n-gram probe on the logs? We run it, and it may well detect exposure to the log text. It cannot tell whether the stored label is what produces the judge's score. Data-level and label-level exposure involve the same text and differ only in the labels, so a text-likelihood probe cannot separate them by construction. Our calibration experiment tests this directly. If perplexity separates the two injection conditions as well as label inertia does, we will report that, and the simpler probe should be preferred.

Why not just evaluate on sessions recorded after the training cutoff? The public satisfaction logs predate every current model, so no post-cutoff split of them exists. Collecting new sessions is the gold standard, and we do collect a small fresh set. However, Test of Time shows that the absence of temporal decay proves little once content is transformed. Fresh logs also cannot be produced at the scale or for the history of tasks that the community benchmarks on. The fresh set serves as an anchor for validating the test, not as a replacement for it.

Why not just paraphrase the queries and documents to decontaminate the benchmark? Test of Time found that transformation cut influence-tracing hit rates from 77.5% to 17.5% while the model kept benefiting from the source material. Paraphrasing therefore makes contamination harder to see, not absent. We include paraphrased copies of injected items in the calibration study. We predict their agreement stays inflated while membership probes go quiet, and that label inertia persists because it does not depend on surface form.

Isn't low responsiveness just the known rigidity of LLM judges? This is the real threat. Siro et al. observed rigidity without any reference to contamination. The design addresses it by comparing the same judge on injected versus held-out items, and on public versus fresh items. Rigidity that does not depend on exposure appears as a low $R$ on both sides and produces no gap. If no gap appears even under label-level injection, the core claim is false. Rigidity is then intrinsic to the judges, and the paper becomes a negative result on contamination together with a cleaner measurement of rigidity.

Proposed Experiments and Decisive Checks

The first check is the calibration study. Label-level injection should raise Spearman agreement on injected items above the held-out items and lower $R$ there under feedback and redundancy edits. Data-level injection should do neither, or do so much more weakly. Both effects should grow with repetition count, and $R$ under neutral edits should stay near zero throughout.

The second check is the audit. For each deployed judge, the proposal succeeds if two things hold: agreement on the public logs exceeds agreement on fresh data drawn from the same sources, and responsiveness is lower on public items than on fresh ones. A judge showing both is drawing on exposure to the benchmark. A judge with a small gap on both measures can be trusted on these benchmarks.

The final check uses USS turn labels with follow-up-turn edits on ReDial and MultiWOZ. It asks whether the inertia pattern transfers from search sessions to dialogue, using the 70→56 human median shift reported by Siro et al. as the reference for how far a responsive judge should move.