When a language model scores well on a public benchmark, we would like to know whether it solved the problems or remembered them. For models post-trained with reinforcement learning (RL), which are rewarded for reaching correct answers, that question has become hard to answer. The standard detectors check whether the model assigns unusually high probability to a benchmark item. RL post-training largely erases that signal. Newer detectors look elsewhere: at how repetitive the model's sampled reasoning is, at how its entropy behaves when asked to critique itself, and at how rigid its hidden states are under paraphrase. Each reaches roughly 0.70–0.79 AUC, where AUC is the probability that a contaminated item scores above a clean one. Each also outputs a single opaque number per item. Published results also disagree about which signal works after which kind of training.
We propose to combine these signals with a shallow decision tree, a model that makes its call through a short chain of human-readable "if feature above threshold" tests. Each threshold is expressed as a percentile of the same signal measured on problems published after the model's training cutoff. Such problems are known to be clean. Every contamination flag then comes with a checkable reason, such as "this item's reasoning diversity is below the 3rd percentile of post-cutoff items, and its hidden states barely move under paraphrase." The hypothesis is that contamination leaves different traces depending on how the model was trained, and a tree can route between detectors in a way a single score or a linear blend cannot. If that holds, the tree should match black-box combiners on accuracy and transfer better to training pipelines it has not seen.
Core Claim
The claim has three testable parts. First, the existing detector signals are complementary in a regime-dependent way, meaning which signal is informative depends on the training stage. A depth-3 or depth-4 tree over percentile-normalized signals should beat the strongest published single detector. It should also come within about 0.02 AUC of a gradient-boosted ensemble. Second, because splits are anchored to a clean reference distribution rather than raw model-specific scales, the tree should lose less accuracy than dense combiners when trained on one contamination pipeline and applied to another. Third, the paths should be stable enough across data resamples that auditors can actually rely on them. If any of these fails, the result is still informative. A linear model matching the tree would mean detectors add up rather than interact. A large gap to boosting would put a price on auditability.
Relation to Prior Work
The motivating evidence comes from four contamination papers.
On The Fragility of Benchmark Contamination Detection in Reasoning Models shows why a single signal fails:
- A reference-model likelihood test (LiRA) catches supervised fine-tuning contamination at 89.1% AUROC.
- One epoch of clean GRPO, an RL method, cuts that by about 14%.
- When the contamination is chain-of-thought data, all ten detectors tested fall to 50–58%.
Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models confirms the likelihood collapse, with perplexity and Min-K% at 0.42–0.53 AUC. Its Self-Critique probe compares entropy sequences of a greedy answer and a self-critique and reaches 0.70 AUC on Qwen2.5-7B-Instruct.
Detecting RLVR Training Data via Structural Convergence of Reasoning measures edit-distance collapse across 32 samples (Min-kNN). It averages 0.70 AUC and reaches 0.76 on a distilled student.
LaRA measures layer-wise hidden-state stiffness. Combined with Self-Critique, it reaches 0.79 ROC-AUC with 31–38% true-positive rate at 5% false-positive rate.
The Fragility paper appears to contradict the other three: it reports near-chance detection where they report 0.70–0.79. We do not assume either side is right. Our pipelines reproduce both settings. The tree's structure, meaning which signal it splits on under which conditions, is the direct test of whether the disagreement reflects different training regimes. None of these papers combines signals across families, explains individual flags, or tests transfer across pipelines. Those are the three things we add.
The method comes from the decision-tree side. Clinical work such as CART-based COPD triage (Fast Prediction of Deterioration and Death Risk…) and leukaemia response modeling (Modelling Predictors of Molecular Response…) motivates the output format. There, shallow trees are used so that non-specialists can execute and challenge a decision. A contested leaderboard claim is a similar situation.
Improving Simple Models with Confidence Profiles (ProfWeight) attaches linear probes to intermediate layers of a network to build per-sample confidence trajectories. It uses them to improve a student CART, lifting accuracy from 74.3% to 87.3% on semiconductor data. We borrow the probe-trajectory idea only as an additional per-item feature family. We do not use it as sample weights, and we do not assume a clean teacher.
DEFT has an LLM propose executable features at each tree node. We test a restricted version as an optional branch, described below. The core proposal does not depend on it, because its per-node LLM cost is high.
Research Program
We first build a per-item feature bank. For every benchmark item and audited model, we compute:
- likelihood scores (loss, Min-K%, Min-K%++, and LiRA where a reference model exists);
- Min-kNN edit-distance statistics over 32 samples;
- the Self-Critique entropy similarity;
- LaRA's paraphrase stability and entity-removal shift, reported per layer band;
- ProfWeight-style confidence trajectories from linear probes on the model's hidden states, with the probe predicting answer correctness.
The key step is reference normalization. Each raw feature is replaced by its percentile within the same model's distribution on a held-out anchor set of clean post-cutoff problems. The PRIME-RL / AIME 2026 set used by LaRA is one example; problems in each benchmark's domain released after cutoff are another. The thresholds a tree learns are then statements like "below the clean 5th percentile." These statements mean the same thing across models, which is what should let rules transfer.
We then fit trees with a fixed depth cap of 3 or 4. We also fit an optimal-depth variant, pruned with cost-complexity pruning (a penalty on tree size). Because standard greedy tree growth is unstable, a known weakness of CART, we report which splits recur across 200 bootstrap resamples. The published audit tree uses only splits that recur in at least 80% of resamples.
In the optional DEFT-style branch, an LLM proposes at most 20 executable features describing the model's response text. An example would be "the final answer appears before any arithmetic." These features enter the bank only if they survive the same bootstrap filter.
Baselines
The naive reference point is chance, 0.50 AUC, along with the likelihood detectors run alone. Perplexity and Min-K% sit at 0.42–0.53 AUC on the RL-MIA benchmark according to the Self-Critique paper. If our pipelines do not reproduce that collapse, our setup differs from the literature and the comparison needs rethinking.
The strongest existing method is LaRA combined with Self-Critique. It reports 0.79 ROC-AUC and 31–38% true-positive rate at 5% false-positive rate on the PRIME-RL / AIME 2026 set (LaRA). It is the strongest competitor because it already combines an internal-geometry signal with a behavioral one. We rerun it with the authors' procedure on the same models, items and anchor data, rather than copying the number. Beating it means higher AUC and higher true-positive rate at 5% false-positive rate on the same split, with a paired bootstrap interval over items that excludes zero. On the 60-sample set, that bar will be hard to clear, so we also report results on our larger controlled suites.
The strongest single-family baselines are Min-kNN (0.70 average, 0.76 on OpenR1-Distill-7B) and Self-Critique (0.70 on Qwen2.5-7B-Instruct, 0.64 on DeepSeek-Math-7B-Instruct), rerun on the same data.
The capacity ceiling is a gradient-boosted tree ensemble on the identical feature bank. It gets the same hyperparameter-search budget as the audit tree: 100 configurations, selected on the same validation split.
Two ablations isolate our ingredients. The first is the same tree grown on raw rather than percentile-normalized features; this tests reference normalization. The second is a tree restricted to one signal family at a time; this tests whether cross-family interaction is doing the work.
Why not just…?
Why not just use sparse logistic regression on the same normalized features? A weighted sum is also auditable. If it matches the tree, we will say so and recommend it. The case for the tree is the regime-dependence visible in the literature. Likelihood signals are strong before RL and near chance after it, per the Fragility paper. Diversity collapse appears specifically after RL with verifiable rewards. That pattern is an interaction, meaning one signal matters only when another takes a certain value, and a linear model cannot express it without hand-built cross terms. The decisive test is a comparison on a pooled training set that mixes pipelines. If logistic regression with pairwise interaction terms equals the tree there, the tree's advantage is not real.
Why not just use gradient boosting and explain it afterwards with feature attributions? This is the most serious threat on accuracy alone. Boosting will likely win in-distribution. After-the-fact attributions, though, do not give a rule an auditor can re-run on a disputed item. Our transfer experiment also tests a specific prediction: many deep splits on model-specific scales should break when the pipeline changes. If boosting transfers as well as the tree, the tree's remaining value is readability alone, and we will report that plainly.
Why not just take the single best detector for each training regime? If the training regime is known, per-regime selection might suffice. Auditors of closed models usually do not know the post-training recipe, however. A tree that detects the regime from anchor-set behavior and routes accordingly is the automated version of this idea. The single-family ablation, run with the true regime given to it as an oracle, serves as the upper bound for this alternative.
Why not just decontaminate the training data with n-gram or embedding matching? Soft duplicates evade this. Soft Contamination Means Benchmarks Test Shallow Generalization finds semantic duplicates for 78% of Codeforces tasks in the Olmo3 corpus, and they still inflate scores. Filtering upstream also cannot audit models whose data is closed.
Proposed Experiments
We build four controlled pipelines where the contaminated items are known.
The first follows the Fragility paper. We contaminate a random half of AIME, AMC, GPQA Diamond, OlympiadBench and Minerva Math items through supervised fine-tuning. We then audit before and after one clean GRPO epoch.
The second is its chain-of-thought contamination setting on DeepSeek-R1-Distill-Llama-8B, where every published detector sits at 50–58%.
The third is the RL-MIA benchmark (AIME 2024/2025, Knights and Knaves, SAT) on Qwen2.5-7B-Instruct and DeepSeek-Math-7B-Instruct.
The fourth is the PRIME-RL / AIME 2026 setup with Eurus-2-7B-PRIME. We add the OpenR1-Distill-7B teacher-prompt audit to test whether rules learned on a teacher survive distillation.
We evaluate three ways:
- In-distribution: AUROC and true-positive rate at 5% false-positive rate, with item-level paired bootstrap intervals.
- Leave-one-pipeline-out transfer: train on three pipelines, test on the fourth.
- Rule stability: the fraction of splits that recur across bootstrap resamples.
For simulatability, three annotators read a flagged item's path and its raw feature values and must reproduce the tree's decision without software help. This measures whether the explanations are executable in practice.
The decisive checks are as follows. The idea succeeds if three things hold together: the audit tree beats the rerun LaRA + Self-Critique baseline on the controlled suites; it stays within 0.02 AUC of boosting in-distribution; and it loses less under leave-one-pipeline-out transfer than both boosting and the raw-feature ablation.
It fails, and the note's claim is wrong, in any of these cases:
- logistic regression ties the tree on pooled data;
- normalization does not improve transfer;
- no pipeline mix lifts chain-of-thought contamination detection above the 58% ceiling.
That last outcome would itself strengthen the Fragility paper's negative result and bound what any combination of current signals can achieve.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
- Detecting RLVR Training Data via Structural Convergence of Reasoning
- LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training
- Fast Prediction of Deterioration and Death Risk in Patients With Acute Exacerbation of Chronic Obstructive Pulmonary Disease Using Vital Signs and Admission History: Retrospective Cohort Study
- Modelling Predictors of Molecular Response to Frontline Imatinib for Patients with Chronic Myeloid Leukaemia
- Improving Simple Models with Confidence Profiles
- Interpretable DNA Sequence Classification via Dynamic Feature Generation in Decision Trees
- Soft Contamination Means Benchmarks Test Shallow Generalization