When a reasoning model has trained on a benchmark question, you might expect it to answer that question with suspicious confidence. After reinforcement-learning post-training, it often does not. Recent work shows that this training erases the probability gap between seen and unseen questions, and that standard detectors fall to near chance on such models. We propose a different signal. Sample many solutions to the same question and look at how they spread out and come back together as generation proceeds. A model that has absorbed a benchmark solution may phrase its opening steps in many ways, yet its attempts should keep collapsing onto the same memorized intermediate states before the answer. A model that is reasoning afresh should branch more like a tree, and its attempts should rejoin, if at all, only at the final answer. That pattern of splitting and merging over generation time is a shape. Zigzag persistent homology, a tool from topology for tracking how clusters appear, split, and merge across a sequence of snapshots, was built to measure exactly that kind of shape.
The core claim is concrete. Pairwise-distance detectors summarize a set of rollouts by how close they are overall. We claim instead that the timing and identity of merge events across reasoning steps separate contaminated from clean prompts in post-trained reasoning models. These are clusters of rollouts that split apart and later rejoin. We claim this signal survives the conditions where likelihood detectors fail, and that it carries enough separating information for audits of fewer than 50 items to be statistically calibrated. We also treat the documentation and integration gap in topology software as a binding constraint on the work rather than an afterthought. The detector will ship as a documented, scikit-learn-style component, so contamination auditors do not need algebraic topology to use it.
What Prior Work Establishes and Leaves Open
Three papers set up the problem. On The Fragility of Benchmark Contamination Detection in Reasoning Models runs two contamination scenarios. In the first, a model is contaminated during supervised fine-tuning and then post-trained with clean GRPO (a reinforcement-learning method); this cuts LiRA's AUROC by about 14% and the loss detector's by about 16%. In the second, contaminating a distilled reasoning model with chain-of-thought data drives 10 detectors down to 50–58% AUROC. This motivates the proposal, and it also constrains it: any signal tied to token likelihood is ruled out. Detecting RLVR Training Data via Structural Convergence of Reasoning is the closest prior work. It observes that rollouts on prompts seen during reinforcement learning with verifiable rewards (RLVR) become rigid and similar. Its detector, Min-kNN Distance, samples 32 completions, computes normalized Levenshtein edit distances between them, and averages the smallest nearest-neighbor distances. It reports an average AUC of 0.70 across 1.5B–32B models and 0.76 on a distillation audit. That detector collapses each rollout to a whole string and the rollout set to one number. It cannot tell "all rollouts are near-copies" apart from "rollouts differ in wording but pass through the same intermediate states", and the latter is the template-reuse pattern we expect after chain-of-thought contamination. When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits supplies the evaluation lens. It shows that detectability scales as $\alpha\rho\sqrt{m}$, where $\alpha$ is the contaminated fraction, $\rho$ is the behavioral separability, and $m$ is the number of audited items. It also shows that Gaussian sample-size approximations fail below about 50 items. A detector is worth more only if it raises $\rho$.
The method comes from topology. Apex Representatives gives an $O(p\cdot m\log m)$ way to track representative cycles through zigzag filtrations, with Dionysus and GUDHI as target libraries. A zigzag filtration is a sequence of spaces connected by inclusions in both directions, so features can both enter and leave. giotto-tda supplies the scikit-learn-compatible pipeline conventions we will follow. TopoBERT applied the Mapper algorithm to BERT embeddings during fine-tuning and found that node purity tracks accuracy (Pearson 0.930 at layer 9). It shows topological summaries of representations can track learning, but it worked on static embeddings of small encoders and never studied memorization. We differ from it in looking at generation-time dynamics across many samples rather than at a single embedding cloud. Its companion version (TopoBERT) also names missing practitioner documentation as an adoption barrier, a point we take seriously.
The Signal
For a prompt $x$, we sample $N$ rollouts and segment each into $T$ reasoning steps. Steps are split at line breaks or step markers and resampled to a common normalized time grid. At each time $t$ we form a point cloud $P_t$ of $N$ step representations. In the white-box variant these are mean-pooled residual-stream activations from a single forward pass over each sampled rollout. In the black-box variant they are sentence-encoder embeddings of the step text. For each time we build a Vietoris–Rips complex $K_t$ on $P_t$ at a scale $\epsilon$ set per model from clean calibration prompts. A Vietoris–Rips complex is the simplicial complex that joins points closer than $\epsilon$. We then compute zigzag persistence over
$$K_1 \hookrightarrow K_1\cup K_2 \hookleftarrow K_2 \hookrightarrow \cdots \hookleftarrow K_T,$$
where each union complex links rollout $i$ at step $t$ to the same rollout at step $t+1$. A connected component therefore follows a bundle of rollouts through time.
From the zigzag barcode we read off three quantities. The first is the time profile of the number of live components. The second is the count and timing of merge events in which components that had separated earlier rejoin before the final step; we call this mid-trajectory reconvergence. The third is the lifetime of the dominant component. The prediction is that contaminated prompts show early splits followed by mid-trajectory merges, a funnel shape, while clean prompts show merges concentrated at the end or none at all. The degree-0 zigzag barcode encodes exactly this split-then-rejoin identity, and a static distance summary discards it. We also keep a simpler time-resolved distance curve as an ablation, because it is the most likely way this idea could turn out unnecessary.
Research Program
The work has three parts. The first is building controlled contamination testbeds. We reproduce both Fragility scenarios: supervised contamination followed by clean GRPO, and chain-of-thought contamination of DeepSeek-R1-Distill-Llama-8B, on the AIME, AMC, and GPQA Diamond audit suite. We add the RLVR setting of the Min-kNN Distance paper. In each case a known half of the benchmark is injected, so the labels are exact.
The second part tests the mechanism behind the predicted signal. We measure whether reconvergence actually occurs, at which layers, and whether it survives paraphrased prompts. A specific risk here is the generalization tension the research file points out: ConStat treats contamination as a failure to generalize to rephrasings, while the Fragility paper shows that chain-of-thought-contaminated models do generalize to similar unseen questions. If our detector also fires on unseen near-neighbors of contaminated items, it detects familiarity with a solution family, not item-level leakage. We will report that outcome rather than hide it.
The third part is software. The detector will be released as a transformer that follows giotto-tda conventions and plugs into the rollouts that existing audits already sample. It will come with a worked tutorial written for evaluation engineers rather than topologists. This responds directly to the adoption gap documented by TopoBERT and by the Guest Editors' Foreword.
Baselines
The naive reference points are random scoring (0.5 AUROC) and plain loss thresholding. The Fragility paper shows loss degrades by about 16% after one GRPO epoch and sits in the 50–58% band in the second scenario, which marks the floor any useful method must clear.
The strongest likelihood baseline is LiRA, a reference-model likelihood-ratio test. The Fragility paper reports 89.1% AUROC for it on supervised-only contamination, falling by about 14% after GRPO. We run LiRA with the same reference models in both scenarios. Our method beats it if it holds AUROC after GRPO, where LiRA has dropped.
The strongest existing structural method is Min-kNN Distance from Detecting RLVR Training Data via Structural Convergence of Reasoning, which reports an average AUC of 0.70. For fairness we compute it from the very same 32 rollouts per prompt that our method uses, tuning $k$ on the same calibration split, so sampling cost is identical. Beating it means a statistically significant AUC gain over its value on our reproduced setups, not merely over the quoted 0.70. Gains must hold in all three contamination settings.
Three ablations isolate the key ingredient. In the first, the embedding Min-kNN ablation, we replace Levenshtein distances with distances between whole-rollout embeddings; this tests whether gains come merely from better representations. In the second, the time-resolved distance-curve ablation, we use the mean pairwise distance at each step $t$, fed to the same small classifier as our barcode features; this keeps time but drops component identity. In the third, the static persistence ablation, we compute degree-0 persistence of whole-rollout embeddings, which keeps topology but drops time. Our claim is supported only if full zigzag features beat all three ablations. It requires both time and component tracking.
Why not just…?
Why not just swap Levenshtein for embedding distances in Min-kNN? This is a serious threat. Static degree-0 persistence is essentially single-linkage clustering, so it carries much of the same information as nearest-neighbor distances. If embedding Min-kNN matches the zigzag detector, then the topology adds nothing and we will say so. The embedding Min-kNN ablation and the static persistence ablation settle this directly. Our hypothesis predicts they will fail specifically in the second scenario, where rollouts differ in wording but share intermediate states, because whole-rollout distances average that structure away.
Why not just track the pairwise-distance curve over steps? This is the strongest simple alternative. A dip in mid-trajectory distance could reveal reconvergence without any homology. The difference is that a distance curve cannot tell one tight cluster that stays tight apart from two clusters that separate and then rejoin. Only component identity across steps distinguishes them. The time-resolved distance-curve ablation tests whether that distinction matters empirically. If it does not, we will recommend the curve, which is cheaper, as the practical tool.
Why not just use rephrased-benchmark performance gaps like ConStat or GSM1k? These methods measure whether scores fall on rephrasings. A Careful Examination of Large Language Model Performance on Grade School Arithmetic found drops of up to 8% for some model families, and ConStat reports a gap of about 8.3% for Mistral-7b on GSM8k. Both are aggregate verdicts and need trusted references or newly written items. The Fragility paper shows chain-of-thought-contaminated models generalize to similar questions, which shrinks exactly the gap these methods rely on. Our method gives per-item scores on the original benchmark. We nonetheless report ConStat's verdicts alongside ours on the same models.
Decisive Checks
The first check is AUROC in both Fragility scenarios and the RLVR setting against every baseline above, under identical rollouts. The second is power at small audit sizes. We estimate separability $\rho$ empirically for each detector and run audits at $m$ between 20 and 50, measuring false-clean rates against the calibration procedure of When Is Benchmark Contamination Detectable?. The detector is useful only if it reaches nominal power where Min-kNN does not. The third is the familiarity control: scores on unseen near-neighbors of contaminated items, generated by rephrasing in the style of MMLU-CF. This tells us whether the signal is item-specific. The fourth is the black-box variant's gap to the white-box one, which decides whether the method can audit closed models. Finally, we check the hypothesis itself. If reconvergence is not more frequent on contaminated prompts, the proposal fails cleanly, and the result still tells the field that structural collapse after RL is a closeness effect, not a branching effect.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Detecting RLVR Training Data via Structural Convergence of Reasoning
- Apex Representatives
- giotto-tda: : A Topological Data Analysis Toolkit for Machine Learning and Data Exploration
- TopoBERT: Exploring the topology of fine-tuned word representations
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Guest Editors' Foreword
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic
- MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark