When a language model has seen benchmark questions during training, its scores overstate what it can do. The usual tests for this "contamination" look at the model's output probabilities and ask whether it is suspiciously confident on test items. Recent work shows these tests break down on reasoning models. Reinforcement learning applied after contaminated fine-tuning erases the confidence gap between seen and unseen questions, and training on worked solutions makes the model about as confident on new, similar questions as on memorized ones. Every output-based detector tested then falls to roughly coin-flip accuracy. We propose to look somewhere these effects may not reach: the shape of the path the model's internal state traces as it reasons step by step. The specific quantity is how sharply that path bends between consecutive steps, which earlier work found tracks logical structure and ignores topic and language.
The two-sentence version is this. Reinforcement learning reshapes what the model outputs, but it is not obvious that it reshapes how the model's hidden states move while it works toward an answer. We test whether curvature-based and non-Euclidean descriptors of that movement still separate contaminated from clean benchmark items after cloaking, and whether they do so better than simple supervised probes on raw activations once topic and difficulty confounds are controlled.
Core Claim
The hypothesis is that contamination leaves a geometric trace in reasoning trajectories that survives the output-space cloaking documented by On The Fragility of Benchmark Contamination Detection in Reasoning Models. That paper motivates the work. It contaminates models through fine-tuning, applies clean reinforcement learning with group relative policy optimization (GRPO), and shows that ten detectors, including Min-K% and LiRA, fall to 50–58% AUROC. It proposes no replacement, and it never examines hidden states.
The prediction has two parts. First, on memorized items the step-to-step trajectory should commit early and bend less, because the answer is effectively retrieved rather than built. Second, items solved by genuine multi-step inference should show the late-layer compression and hierarchical structure that flat probes fail to capture. The value of a geometric statistic, rather than a raw activation classifier, is invariance. Second-order curvature is reported to ignore topic and language. An audit built on it should therefore respond to how an item was solved and not to which item it is. That property matters because real audits always compare benchmark items against fresh items that differ in content.
What Prior Work Supplies and Leaves Open
Two recent papers supply the measurement tools, and neither tests contaminated models. The Geometry of Reasoning: Flowing Logics in Representation Space treats the sequence of hidden states across reasoning steps as a trajectory. It finds that absolute position clusters by topic, while Menger curvature, the inverse radius of the circle through three consecutive points, clusters by logical structure and is destroyed when steps are shuffled. The quantity, for step states $x_{t-1}, x_t, x_{t+1}$, is
$$\kappa_t = \frac{2\sin\angle(x_{t-1},x_t,x_{t+1})}{\lVert x_{t-1}-x_{t+1}\rVert}.$$
We reuse this measure directly. Hyperbolic Geometry of Reasoning: Probing LLM Hidden States trains structural probes that map hidden states into either flat space or the Poincaré ball, a model of hyperbolic geometry in which volume grows exponentially and trees fit naturally. On PrOntoQA it finds that the flat probe on DeepSeek-R1's layer 27 drops to Spearman 0.488 with distortion 0.562, while the hyperbolic probe holds at 0.967 with distortion 0.090. The flat probe stays healthy on instruction-tuned Qwen2.5, at 0.955. That gap between reasoning-trained and instruction-tuned models suggests the gap between flat and hyperbolic distortion carries information about how a model computes, not only about probe design. We test that suggestion on contaminated items. The volume argument for why trees crowd together in flat space comes from Exploiting Data Geometry in Machine Learning. The learnable product of hyperbolic, spherical and flat components comes from Mixed-curvature Variational Autoencoders. We use that product only as an ablation, because the evidence for it here is weakest.
One paper constrains the design. Non-Euclidean Geometries for Prototype-Based Image Classification: a Reality Check reports that curved geometries often fail to beat high-dimensional flat baselines on real data. Accordingly, the proposal makes no assumption that hyperbolic features win. Flat probes appear as first-class competitors and as the ablation that decides the question.
The closest existing detector is RADAR. It classifies recall versus reasoning from 37 internal features, including attention specialization and the layer at which confidence converges, and reaches 93.0% accuracy on a 100-item test set. RADAR differs from this proposal in three ways. Its features are flat and heuristic rather than geometric invariants. Its dataset is small and curated. It was never run against models cloaked by reinforcement learning. This proposal asks the question RADAR could not: does any internal signal survive the specific cloaking that defeats output detectors?
Research Program
We build contaminated models with known ground truth, reusing the protocol of the Fragility paper. Each benchmark (AIME, AMC, GPQA Diamond, OlympiadBench, Minerva Math) is split at random, at the item level, into members and non-members. Because the split is random, the two groups share the same distribution and any detection reflects contamination rather than content shift. In Stage I, a base model is fine-tuned on member items and then trained with clean GRPO, and we save checkpoints before reinforcement learning, after one epoch, and later. In Stage II, DeepSeek-R1-Distill-Llama-8B is fine-tuned on worked chain-of-thought solutions to member items.
We add a controlled testbed using items generated by DyVal. DyVal builds reasoning problems from directed acyclic graphs, which gives each item an exact ground-truth depth for each step. On these items we can contaminate a model and also train depth probes, which the math benchmarks do not allow.
For each item and each layer, we segment the model's chain of thought into steps. Following the hyperbolic paper's finding, the state at each step is read at "thinking tokens" such as "Therefore," because these recover depth far better than averaging over all tokens (0.871 versus 0.390). From each trajectory we compute three label-free descriptors. The first is the profile of Menger curvature along the steps. The second is the profile of step-to-step velocity. The third is the layer at which the trajectory's direction stabilizes. On the DyVal and PrOntoQA testbeds we also train flat and hyperbolic depth probes on clean models only, then measure each probe's distortion on member and non-member items. The resulting contamination descriptor is the distortion gap, which is defined as follows:
$$\Delta_{\text{dist}} = D_{\text{flat}} - D_{\text{hyp}},$$
where $D_{\text{flat}}$ and $D_{\text{hyp}}$ are mean absolute errors between probe distances and true depth gaps. A small detector, logistic regression on these low-dimensional statistics, produces an item-level score. We also form a benchmark-level two-sample test against fresh DyVal items, which is the realistic audit setting.
The method requires access to hidden states. It applies only to audits of open-weight models or to audits run by a model's developer, not to closed APIs. We state this as a scope limit.
Baselines
The naive reference point is chance, 50% AUROC. It matters because the Fragility paper reports that reference-free output detectors reach about 50% after GRPO, and that almost all ten detectors average 50–58% AUROC in Stage II. Any internal method must clearly exceed that band to be worth hidden-state access.
The strongest output-space method is LiRA, a likelihood-ratio membership test that compares the target model against reference models. The Fragility paper reports 89.1% AUROC for LiRA before reinforcement learning and a drop of about 14% after one GRPO epoch. We rerun LiRA, Min-K% from Detecting Pretraining Data from Large Language Models, and the plain loss test on the same checkpoints and the same member/non-member splits. This reproduces the collapse in our own setup rather than relying on the reported numbers.
The strongest internal competitor is a supervised flat probe: logistic regression and a small multilayer network trained on raw pooled hidden states from every layer. These probes get the same labeled training items, the same layers, and the same hyperparameter search budget as our geometric detector. Next to them we run RADAR's 37-feature ensemble, reimplemented and trained on the same items. No reported result found on Lacuna for RADAR or raw-state probes on the reasoning contamination suite, and RADAR's 93.0% comes from its own 100-item set, so it is not directly comparable.
The ablation that isolates the key ingredient replaces second-order curvature with zeroth-order position statistics computed on the same thinking-token states. A second ablation shuffles the step order, which should destroy any curvature signal that is genuine. A third swaps the hyperbolic probe for a mixed-curvature product probe.
We count the proposal as beating the field under three conditions. It must reach at least 70% AUROC on post-GRPO and Stage II checkpoints, where output detectors sit in the 50–58% band. It must match or exceed the raw-state probe in-distribution. It must clearly beat that probe when trained on one benchmark and tested on another.
Why not just…?
Why not just run Min-K% or LiRA? The Fragility paper has already settled this for our setting. Clipping in GRPO's importance sampling closes the probability gap, and contamination with worked solutions makes the model equally confident on unseen items. We include both detectors to confirm the collapse on our checkpoints, but the published evidence says they fail exactly where this audit is needed.
Why not just train a linear probe on raw activations? This is the real threat. A probe trained on labeled members and non-members from the same benchmark may well match the geometric detector, because a random split removes confounds and raw states contain everything the curvature statistics contain. Our answer is that audits rarely have in-distribution labels; they compare old benchmark items against fresh ones that differ in topic. The geometry paper's evidence that position tracks topic and curvature tracks logic predicts that raw probes will latch onto content and fail under cross-benchmark transfer. If raw probes transfer just as well, the geometric detector adds nothing beyond interpretability. We would report that outcome as a negative result, and the decisive check below is designed to expose it.
Why not just perturb the questions and look at the accuracy drop? Rephrasing and shuffling, as in MMLU-CF, reveal benchmark-level inflation: about a 14% average drop across more than 40 models, and o1 falling from 92.3% to 80.3%. This approach is a strong behavioral competitor, and we include a per-item version in which an item is scored by the change in correctness across perturbed variants. Its weakness is the same generalization the Fragility paper documents. A model trained on worked solutions transfers its heuristics to rephrasings, so perturbation may not flag it. The experiment settles this directly: we compare per-item perturbation AUROC against geometric AUROC on the Stage II model.
Proposed Experiments and Decisive Checks
The first check is the survival curve. We plot AUROC for every detector against GRPO training steps in Stage I. The proposal succeeds only if the geometric curve stays flat while the LiRA and loss curves fall.
The second check is transfer. We train each internal detector on members and non-members from AIME and AMC, then test on GPQA Diamond and OlympiadBench. In a parallel test, each detector must separate contaminated DyVal items from fresh DyVal items of matched graph depth and width. Matching depth and width removes difficulty as an explanation.
The third check concerns mechanism. On the DyVal and PrOntoQA testbeds we ask whether the distortion gap $\Delta_{\text{dist}}$ shrinks for member items, which would mean memorized items skip the hierarchical compression that flat probes cannot capture. We also confirm that shuffling steps removes the curvature signal.
The fourth check is a stress test against a misleading signal. We contaminate with paraphrased worked solutions, and with solutions to structurally similar but different problems. A detector that fires on both is measuring "familiar reasoning pattern" rather than "seen item," and we report that distinction explicitly.
A clear negative outcome would be informative in its own right. If curvature descriptors also collapse after GRPO, the cloaking reaches inside the model's computation and not just its outputs. That would bound what any white-box audit can recover. If raw probes match geometry under transfer, the field can drop the Riemannian machinery for this task.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- The Geometry of Reasoning: Flowing Logics in Representation Space
- Hyperbolic Geometry of Reasoning: Probing LLM Hidden States
- Exploiting Data Geometry in Machine Learning
- Mixed-curvature Variational Autoencoders
- Non-Euclidean Geometries for Prototype-Based Image Classification: a Reality Check
- RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Detecting Pretraining Data from Large Language Models
- MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark