Auditing Hidden Contamination with Hyperbolic Structural Probes

Reasoning benchmarks are leaking into training data. The usual way to catch this is to check whether a model finds a test question suspiciously easy to predict, meaning its loss on that text is unusually low. That check stops working once a model has been through reinforcement-learning post-training: On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that a single epoch of clean policy optimization drives loss-based detectors to about 50% AUROC, which is chance. We propose to look inside the model instead. When a model really solves a multi-step problem, its hidden states should contain the problem's reasoning structure: which intermediate quantities depend on which, and how deep each one sits. When it has memorized the answer, it does not need to build that structure.

The tool is a structural probe, a single linear map from hidden states into a space where distances are supposed to reproduce the problem's dependency graph. We send the probe into hyperbolic space, a curved geometry whose volume grows exponentially with radius the way trees do. The probe is trained only on freshly generated problems that the model cannot have seen. Each test instance is then scored by how badly its internal geometry departs from what clean problems of the same shape produce. This score is not a likelihood statistic, so the argument that clipping in policy optimization erases loss gaps does not automatically apply to it. Whether the score survives that training is the empirical question at the center of this proposal.

Core Claim

On procedurally generated reasoning graphs with controlled, known contamination, a linear hyperbolic structural probe fit only on clean instances yields a per-instance "structural fidelity" score. We hypothesize that this score separates contaminated from uncontaminated instances after reinforcement-learning post-training, in the regime where loss, Min-K%, and likelihood-ratio detectors have collapsed. It should also do so among instances the model answers correctly, so the score is not just a proxy for being right.

The closest prior work is Hyperbolic Geometry of Reasoning: Probing LLM Hidden States. It trains linear probes into the Poincaré ball on PrOntoQA's linear deduction chains and finds that they recover reasoning depth in DeepSeek-R1's last layer (Spearman ~0.967) where Euclidean probes degrade (0.488). That work is purely diagnostic. It never asks whether the recovered geometry differs between memorized and genuinely solved instances, and it never leaves chain-shaped problems. We reuse its probe as the measuring instrument and turn it into an auditing statistic on branching graphs.

Research Program

The three directions play different roles here. Benchmark contamination is the problem being solved. Hyperbolic embedding supplies the target geometry. Linear representational mapping, the brain-encoding tradition, supplies the methodological discipline. Mapping Brains with Language Models: A Survey of Neural Alignment argues that the map from network activations to a target must be linear, because a flexible nonlinear map can manufacture correspondence that is not in the representation. The same logic applies to auditing: an MLP probe could "find" structure in any hidden state, so a fidelity gap would mean nothing. We also borrow that field's protocol of a ridge-regularized fit, evaluation on held-out stimuli, and a layer-by-layer sweep. Abstraction Induces the Brain Alignment of Language and Speech Models finds that abstraction peaks in intermediate layers before task-specific compression in late layers. That finding tells us not to assume the final layer is the informative one.

The problems come from DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks. It generates arithmetic, logic, and reachability problems as directed acyclic graphs with controllable depth and width across four difficulty tiers. It reports that models strong on static benchmarks score near zero on these problems and that depth hurts more than width. DyVal treats models as black boxes; we use its generator for two different purposes. First, the generator gives a ground-truth graph for every instance. Second, members and non-members can be drawn from exactly the same distribution, which removes the distribution-shift confound that inflates many membership-inference results.

The probe works as follows. For each graph node $i$, let $h_i$ be the layer-$\ell$ hidden state at the token that names node $i$. The probe maps $h_i$ to $z_i = \exp_0(W h_i)$ in the Poincaré ball, where $W$ is a learned linear matrix and $\exp_0$ is the fixed exponential map at the origin. It is fit on clean instances by minimizing the mismatch between hyperbolic distances and graph distances, with ridge regularization on $W$:

$$\mathcal{L}(W) = \sum_{i,j} \big| d_{\mathbb{B}}(z_i, z_j) - d_G(i,j) \big| + \lambda \|W\|_2^2 ,$$

where $d_{\mathbb{B}}$ is the Poincaré geodesic distance and $d_G$ is the undirected shortest-path distance in the DyVal graph. For a test instance, the raw distortion $D$ is the mean absolute distance error. The audit score is $D$ standardized against the distribution of $D$ on fresh clean instances with the same depth, width, and correctness. We pre-register lower fidelity in members as the primary hypothesis, on the reasoning that recalled answers need no internal graph. The opposite outcome, unusually crisp geometry from rehearsed traces, is plausible enough that we also report the two-sided version of the score.

Contamination follows the two-stage protocol of the Fragility paper, so its detector numbers serve as a reference. In Stage I, Qwen2.5-7B-Instruct is fine-tuned on a member set of DyVal instances with their answers, then trained with group-relative policy optimization on clean, disjoint DyVal problems. In Stage II, DeepSeek-R1-Distill-Llama-8B is fine-tuned on members' chain-of-thought traces. In both stages we also train contamination-free twins under identical procedures, which lets us separate the effect of contamination from the effect of post-training itself. PrOntoQA, using Raj's setting, is a sanity replication on chain-shaped problems.

Baselines

The naive reference point is chance at 50% AUROC. A second naive reference is the DyVal correctness gap: the drop in accuracy between members and fresh instances, aggregated over a set rather than scored per instance. This tells us how much contamination signal is visible from the output alone.

The strongest existing per-instance detector is the likelihood-ratio attack LiRA, together with Loss and Min-K%, as run in On The Fragility of Benchmark Contamination Detection in Reasoning Models. That paper reports 89.1% AUROC for LiRA after contaminated fine-tuning, roughly 50% after one epoch of clean policy optimization, and an average of 50–58% across ten detectors under chain-of-thought contamination. We rerun all three detectors on our own DyVal member/non-member splits, with the same models and the same reference-model budget. No reported result was found on Lacuna for these detectors on DyVal specifically. Beating them means AUROC of at least 70% and clearly higher true-positive rate at 5% false-positive rate after policy optimization, with bootstrap confidence intervals that do not overlap.

The ablation that isolates geometry is an otherwise identical Euclidean structural probe. It has the same linear map, regularization, training data, and per-layer hyperparameter search, and differs only in dropping $\exp_0$ and using Euclidean distance. A second ablation fits the hyperbolic probe to node depth alone and ignores pairwise distances. This tests whether branching structure, as opposed to depth, carries the signal.

The strongest "no structure" competitor is a supervised linear membership classifier on the same hidden states. Because a real auditor does not have member labels for the model under audit, this classifier is trained on a separately contaminated reference model, the same assumption LiRA makes, and then transferred to the audited model.

All probes and classifiers receive the same number of training instances, the same layer sweep, and the same tuning grid.

Why not just…?

Why not just use a Euclidean probe? It may well suffice in intermediate layers. Raj's Euclidean collapse is a late-layer, reasoning-model phenomenon (0.488 versus 0.967 at layer 27 of DeepSeek-R1). On branching graphs, Poincaré Embeddings for Learning Hierarchical Representations reports Euclidean distortion so large that 200 dimensions reach only 0.168 MAP on WordNet, against 0.823 for five hyperbolic dimensions. That argues for curvature, but it concerns learned embeddings, not probes of a fixed model. The matched Euclidean ablation decides the question. If it audits equally well, we will report geometry-agnostic structural fidelity as the contribution and drop the hyperbolic claim.

Why not just train a linear classifier on activations to predict membership? This is the most serious threat. If contamination leaves any linearly readable trace, a direct classifier is simpler than a structural probe. The structural score has two possible advantages: it needs no member labels, only fresh generated problems, and it gives an interpretable reason for each flag. The reference-model-transfer baseline measures this directly. If the transferred classifier matches the structural score, the label-free property is the only remaining advantage, and we will say so.

Why not just evaluate on DyVal-style dynamic problems and skip detection? Dynamic benchmarks measure capability without contamination, but they cannot say whether a reported static score was inflated. As the Fragility paper shows, chain-of-thought contamination transfers heuristics that also help on related fresh problems, so a capability gap understates leakage. Detection and dynamic evaluation answer different questions. Our experiments use the dynamic generator as ground truth for building a detector, not as a replacement for one.

Why not just use perplexity or likelihood-ratio tests? The Fragility paper already shows they reach 50–58% AUROC in exactly our regime. We rerun them only to confirm that collapse on our splits.

Proposed Experiments

The decisive experiment is Stage I after policy optimization. We compute AUROC of the hyperbolic fidelity score against LiRA, Loss, and Min-K% on distribution-matched DyVal members and non-members, restricted to instances the model answers correctly, and sweep layers. If no layer beats LiRA by a clear margin, the core claim fails.

The twin comparison is the second check. Raj observed late-layer compression in DeepSeek-R1: a 41% drop in activation norm and a 43% drop in participation ratio, a measure of effective dimensionality. If the fidelity distribution of contamination-free twins shifts after policy optimization in the same way as that of contaminated models, the signal reflects the training procedure rather than memorization.

Stage II tests the boundary case named in the Fragility paper, where contaminated chain-of-thought produces generalizable heuristics. A null result there is informative: it would suggest that trace contamination leaves the model actually reasoning, and we will report it as a limit of the method rather than hide it.

Finally, we stratify results by DyVal depth and width. DyVal found that depth drives failure, so we test whether the audit signal concentrates in deep graphs. We also replicate on PrOntoQA to check that the probe reproduces Raj's correlations before any contamination is introduced.