Behavioral Sibling Tests for Contamination That Survives RL Post-Training

When a language model has seen a benchmark's test questions during training, its score on that benchmark overstates what it can do. Most current detectors look for traces of that exposure in the model's token probabilities, such as unusually low loss on seen text. That signal can be erased. Reinforcement-learning post-training flattens the probability gap between seen and unseen questions while keeping the inflated accuracy, which pushes detectors to near chance. The inflated accuracy is the one thing such post-training cannot remove, because it is the reason anyone contaminates. We propose to detect contamination from accuracy alone. For each benchmark item we generate "siblings": versions with the same reasoning structure but new numbers, names, and therefore a new answer. We then ask whether the model solves the original item more often than its sibling success rate and its estimated ability predict. A memorized answer does not carry over to a sibling whose answer is different.

The second half of the proposal comes from data-efficiency analysis, the study of how much data is enough for a stable conclusion. It asks how many siblings per item, and how many items, are needed before such a test is reliable. We give sample-size rules for both per-item flags and benchmark-level inflation estimates. We also describe an adaptive allocation scheme that spends siblings only where an item looks suspicious, in the spirit of adaptive testing. The result is a contamination audit that needs no log-probabilities, works on black-box APIs, and comes with stated sample budgets.

Core Claim

Contamination that inflates accuracy leaves an item-specific success bonus. That bonus is visible only when the model's success on an item is compared against its success on structurally matched items. A hierarchical item-response model can estimate it with a known number of queries. The model is a version of item response theory (IRT), the psychometric approach that predicts the chance of a correct answer from a test-taker's ability and an item's difficulty. Because the bonus is measured in correctness rather than in likelihood, it should survive the reinforcement-learning cloaking that defeats loss-based detectors.

Concretely, let $\theta$ be the model's latent ability. Let $b_i$ be the difficulty of item family $i$, which is shared by the original and its siblings. Let $d_{ij}$ be a small per-sibling difficulty offset. Let $\gamma_i \ge 0$ be the memorization bonus that applies only to the original. The model is

$$ P(\text{correct on sibling } j \text{ of } i) = \sigma(\theta - b_i - d_{ij}), \qquad P(\text{correct on original } i) = \sigma(\theta - b_i + \gamma_i), $$

where $\sigma$ is the logistic function. Item $i$ is flagged when the posterior mass on $\gamma_i > 0$ exceeds a threshold. The benchmark-level inflation estimate is the average gap between observed original accuracy and the accuracy implied with $\gamma_i = 0$.

A plain binomial argument sets the data-efficiency question. Suppose the model solves the original nearly always and solves siblings at rate $p$. Then about $\log(1/\delta)/\log(1/p)$ sibling failures are needed to reject "no bonus" at error $\delta$. Hard items, where $p$ is small, therefore need only a few siblings. Easy items, where $p$ is near one, are expensive and also uninformative, since an item the model can genuinely solve cannot reveal memorization. Estimating benchmark-level inflation within $\epsilon$ needs on the order of $\sigma^2/\epsilon^2$ items by the usual variance argument. Pooling difficulty across families is what shrinks $\sigma^2$.

Relation to Prior Work

Two results motivate the proposal by showing that the dominant signals break. On The Fragility of Benchmark Contamination Detection in Reasoning Models contaminates models through supervised fine-tuning and then applies GRPO, a reinforcement-learning method. LiRA, a likelihood-ratio membership detector, reaches 89.1% AUROC (area under the ROC curve) before reinforcement learning. One epoch of GRPO cuts LiRA's AUROC by about 14%, pushes reference-free detectors to about 50%, and leaves the 8.8% pass@1 inflation in place. That surviving accuracy inflation is the signal we target.

Test of Time shows that the post-cutoff accuracy drop disappears when questions are rephrased. Influence-function attribution also falls from 77.5% to 17.5%. This constrains our design: siblings must not be paraphrases, because a paraphrase keeps the same answer and can carry over the benefit of prior exposure. Our near siblings change the answer.

Two papers supply the method. Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective uses IRT-based computerized adaptive testing and reports an 80% reduction in items with 0.90+ rank correlation. It also proposes the IRT guessing parameter $c$ as a leakage flag. We reuse its adaptive-selection idea, but $c$ is a property of an item estimated across test-takers. It cannot say whether this particular model memorized this particular item, and our per-model, per-item $\gamma_i$ can. DyVal generates reasoning problems from directed acyclic graphs with controllable depth. We use it for "far" siblings and as a guaranteed-clean control set. DyVal fixes 500 samples per task without a sufficiency analysis, which our sample-size rules supply.

Three further methods are competitors rather than components. DCR adjusts accuracy with a heuristic fuzzy-logic factor, $Acc_{adj} = Acc \times (1 - \text{DCR Factor})$. It reports adjusted scores within 4% of uncontaminated baselines, but it gives no sample-size guarantee and no per-item flags. The Sharded Rank Comparison Test in Proving Test Set Contamination in Black-Box Language Models works with 1,000 examples but cannot see contamination if the items were shuffled before training. Polarized Augment Calibration perturbs words and measures certainty, so it remains a likelihood-based method.

Research Program

We will build sibling generators for three item types. For grade-school and competition math (GSM8K, and AIME/AMC-style items as used in the fragility study), we will extract a symbolic solution template. We will then produce near siblings by resampling constants and entity names, recomputing answers by execution, and keeping only siblings whose solution graph matches the original. For code (HumanEval and LiveCodeBench problems), near siblings change specification constants and regenerate the unit tests. Far siblings for math and logic come from DyVal graphs matched in depth and width. Generators are validated by checking that a clean reference model's accuracy on originals and siblings agrees within sampling error.

The hierarchical model above is fit per model with weakly informative priors. Sibling difficulty offsets $d_{ij}$ are learned from a panel of clean reference models, so that a genuinely harder sibling is not read as evidence of memorization.

On top of this sits a sequential allocation rule. Each item starts with one original query and one sibling query. Further siblings go only to items whose posterior on $\gamma_i$ remains uncertain. Querying stops once the posterior settles or the budget is exhausted. We will derive and empirically check the per-item and per-benchmark sufficiency thresholds, and report AUROC and inflation-estimate error as functions of total queries.

Baselines

The do-nothing reference is raw accuracy on the original benchmark, which overstates ability by exactly the inflation we want to measure.

The simplest serious competitor is sibling-only accuracy: report mean accuracy on regenerated items and call the gap from the original the inflation, as EvoEval-style rewriting does. The survey Benchmark Data Contamination of Large Language Models: A Survey reports a 39.4% average drop for EvoEval across 51 models. It runs on exactly the same sibling pool and query budget as our method, so any advantage we show comes from the modelling and the allocation, not from extra data. We beat it if our inflation estimate is closer to the known injected inflation at equal queries, and if it gives per-item flags that it cannot produce.

The strongest existing detectors on this question are LiRA and Min-K%, run as in the fragility study. On the same contaminated-then-GRPO checkpoints, the reported numbers are 89.1% AUROC for LiRA before reinforcement learning, and near 50% (reference-free) or 50–58% (Stage II, across all ten methods) after it. We give these detectors white-box log-probability access, which favors them, and tune their reference models on the same held-out clean split. Beating them means per-item AUROC of at least 0.80 on post-GRPO checkpoints, where they sit near chance.

We also run the IRT guessing-parameter flag from the adaptive-testing paper and DCR on the same items. For the Sharded Rank Test, we include a shuffled-ingestion condition, where it is expected to fail by construction, and a canonical-order condition, where it should succeed.

Two ablations isolate our ingredients. The first replaces hierarchical pooling with a raw per-item difference between original and sibling accuracy. The second replaces adaptive allocation with uniform siblings at the same total budget. If the raw-difference ablation matches the full model, the IRT machinery is unnecessary and we will say so.

Why not just…?

Why not just report accuracy on regenerated items and stop there? For a single benchmark-level number, this may be nearly as good as our method, and we treat it as a real threat. Its weaknesses are that it gives no per-item evidence, has no stated sample budget, and charges sibling difficulty drift to contamination. The equal-budget comparison above settles whether pooling and difficulty calibration matter. If they do not, the contribution narrows to the sufficiency analysis and the per-item test.

Why not just use post-cutoff temporal splits? Test of Time shows the decay disappears under rephrasing: rephrased questions score +2.19 points post-cutoff on physics, while cloze versions of the same papers drop 3.83 points. Perturbing LiveCodeBench problems likewise removed its post-cutoff decay. Temporal splits also say nothing about items released before the cutoff.

Why not just use membership inference on log-probabilities? The fragility study shows these detectors collapse to 50–58% AUROC after reinforcement-learning post-training, and black-box APIs often hide log-probabilities. Our method needs only answers.

Why not just paraphrase the benchmark? A paraphrase keeps the same answer, so it can launder exposure rather than expose it, as the influence-attribution drop in Test of Time shows. Near siblings change the answer, which is precisely what breaks a memorized response. We test this directly by running our model with paraphrase siblings substituted for near siblings and expect the detection AUROC to fall.

Why not just use the IRT guessing parameter? $c$ is fit per item across many models. A memorized item looks the same as an easy-to-guess one, and the parameter cannot single out which model memorized it. Our comparison with $c$ on identical items measures this gap directly.

Proposed Experiments

The decisive experiment reproduces the fragility study's two contamination stages with known ground truth. We contaminate a random half of GSM8K test items and of the competition-math items through supervised fine-tuning at 1, 4, and 16 exposures, then apply clean GRPO. Stage II uses a DeepSeek-R1-Distill-Llama-8B-scale model fine-tuned on chain-of-thought traces for the contaminated items. We report per-item AUROC, benchmark inflation-estimate error, and the full queries-versus-AUROC curve. The method succeeds if its AUROC stays at or above 0.80 after GRPO while LiRA and Loss fall as reported. It fails if $\gamma_i$ also shrinks under GRPO, which would mean reinforcement learning turns memorized answers into transferable skill.

Stage II carries an honest risk. That paper reports these models "internalize reasoning structures." If the 11.7% boost carries over to near siblings, our test will report little item-specific inflation. We would present that as a finding: the gain is transferable, and a sibling-level evaluation is the right way to measure it.

Two further checks bound the method. On DyVal-generated originals, which no model can have seen, the false-positive rate should match the nominal $\delta$. On public models, we run audits on GSM8K and HumanEval to show black-box use, without ground truth, and compare our verdicts against the Sharded Rank Test and DCR verdicts on the same models. The sufficiency claim passes if the empirical number of siblings needed per item tracks the $\log(1/\delta)/\log(1/p)$ rule across difficulty bins. It also requires adaptive allocation to reach the uniform-allocation AUROC with clearly fewer queries.