Auditing Benchmark Contamination by Testing the Law of Generation Paths

When a language model has seen a benchmark during training, it does not only answer those questions more accurately. It also answers them differently: its reasoning becomes rigid, its confidence evolves along a narrower route, and repeated samples look alike. Current detectors capture pieces of this. Some score one sequence of token probabilities. Some compare sampled answers with string edit distance. Some inspect hidden activations. None treats the object that actually changes, which is the probability distribution over whole generation trajectories. We propose to treat each generation as a path. The path is the curve traced by per-token uncertainty features, or by step-level embeddings when only text is available. We then ask a statistical question: is the distribution of paths on a suspect benchmark different from the distribution on freshly generated, matched control items? We answer it with a two-sample test built on the signature of a path. The signature is the sequence of iterated integrals that summarizes a curve's ordered shape while ignoring how fast it was traversed. The test returns a calibrated p-value for the whole benchmark, needs no labeled contaminated examples, and is unaffected by verbosity or pacing by construction.

The theory of stopped processes is what makes this more than a feature swap. Reasoning traces stop when they hit an answer boundary, such as a boxed answer or an end token. For stopped Brownian motion, the naive expected signature has a finite radius of convergence. That rules out the classical argument that it pins down the path distribution (Boedihardjo et al.; Li & Ni). A contamination test built on raw signature moments therefore has no guarantee of detecting a difference between stopped rollout distributions. The robust, tensor-normalized signature moments of Chevyrev & Oberhauser avoid this problem because they are bounded and characteristic without any growth condition. Characteristic here means the resulting distance between distributions is zero only if the distributions are identical. The proposal applies this validated tool, and tests empirically whether the theoretical distinction matters on real reasoning traces.

Core Claim

A permutation test using maximum mean discrepancy (MMD), a kernel distance between two sample distributions, computed with the robust signature kernel on generation paths, will do three things. First, it will detect benchmark-level contamination at lower sampling cost than edit-distance rollout comparison. Second, it will stay valid under paraphrase and step insertion. Third, it will work under reinforcement learning with verifiable rewards (RLVR), where likelihood detectors fail. We also claim that the robust normalization, which the stopped-process theory prescribes, has measurable power consequences compared with unnormalized signatures on terminated traces. If path order carries no signal beyond summary statistics, the second part of the ablation below will show it and the claim fails.

What Exists and What Differs

Three detectors motivate the gap.

  • MIN-K% PROB averages the lowest-probability tokens of a single text. It is known to degrade under paraphrase.
  • Min-kNN Distance shows that RLVR makes seen prompts produce near-identical rollouts. It measures this with normalized Levenshtein distance over 32 completions, at roughly 6.65 seconds per item.
  • LOS-Net (also here; Learning on LLM Output Signatures for Gray Box LLM Behavior Analysis) trains a small supervised Transformer on sequences of top-K token distributions. Its "signature" is only a name, with no iterated integrals and no invariance guarantee.

A Narrowing Geometry in Contaminated Reasoning supplies the mechanistic motivation. It finds that contaminated reasoning concentrates in a few activation directions, but its analysis is static, per-layer, and white-box.

Our method differs from all four in the same way. It compares distributions of ordered trajectories rather than scoring single sequences or unordered strings. It needs no labels. It reports a hypothesis-test p-value rather than a threshold score.

The method comes from two sources. Chevyrev & Oberhauser supply the robust signature kernel and its characteristic property. Randomized signatures supply a cheap random-projection surrogate, costing $O(k^2 d)$ per step for a $k$-dimensional reservoir. That surrogate is what makes high-dimensional embedding paths tractable.

Research Program

We build two path encodings.

  • Gray-box path. At each generated token we record four quantities: entropy, top-1 probability, top-1 minus top-2 margin, and the log-rank of the chosen token. Together these give a 4-dimensional curve. It uses exactly the information LOS-Net shows is missing from scalar token probabilities.
  • Black-box path. The reasoning trace is split into steps, each step is embedded with a frozen sentence encoder, and the embeddings are joined into a curve. Signatures of this curve are computed through the randomized-signature reservoir.

Each path stops at the answer boundary. We optionally add a time channel. Adding time breaks invariance to reparameterization, and we ablate its effect rather than assume it.

The audit proceeds as follows. We draw $N$ rollouts on each item of a suspect benchmark subset. We draw the same number on a matched control set: freshly generated items of the same template and difficulty, such as new MSCalc-style problems. We then compute the robust signature MMD between the two path collections and calibrate it by permutation. The same kernel also gives a per-item score: the mean within-item signature distance among the $N$ rollouts. This is a direct continuous analogue of Min-kNN Distance, so the two can be compared head to head.

The theoretical-validation strand has two parts:

  • Synthetic sanity check. On Brownian motion stopped at a disk, where the finite-radius result is known, we confirm that the robust MMD separates two stopped laws that differ only at high signature levels, and that unnormalized truncated signatures do so less reliably.
  • Transfer to real traces. We check whether the same gap appears on real traces that terminate early versus late.

Baselines

The naive reference is a length-and-accuracy test. We compare mean generation length and accuracy between the suspect and control sets with a two-sample t-test. Contamination often shows up as shorter, more accurate answers, so beating this rules out the trivial explanation. Beating it means higher power at equal set size, with calibrated false-positive rate on clean splits.

The strongest black-box competitor is Min-kNN Distance, which reports average per-item AUC 0.70 across 1.5B–32B RLVR models. Its AUC falls only from 0.72 to 0.71 under paraphrase, and it reaches 0.76 on the OpenR1-Distill-7B distillation audit. We run it on the same models, prompts, and rollouts. We sweep $N \in \{4, 8, 16, 32\}$ for both methods, so the comparison is at equal generation budget. It counts as beaten if our per-item score matches its AUC with a quarter of the rollouts, or exceeds it at $N=32$, and if our set-level test reaches detection with fewer items.

The strongest gray-box competitor is LOS-Net, which reports 0.95 AUC on BookMIA with Llama3-30b (source). Because it is supervised, we give it the same held-out labeled split it would need. We then evaluate it both in-distribution and after a shift in reasoning format, such as a different chain-of-thought template, where invariance should matter.

For MIN-K% PROB, the file gives relative gains: 7.4% AUC on WikiMIA and 12.2% on leaked benchmarks (Shi et al.). It reports no absolute AUC, so we rerun it rather than quote it.

White-box ECI from Xie et al. serves as an upper-reference diagnostic on open models. It is not a competitor, because it needs access the audit setting lacks.

The key ablations isolate path order. The first replaces the signature kernel with a Gaussian-kernel MMD on unordered per-trace summary statistics (mean, variance, and quantiles of the same four channels). The second shuffles token order before computing signatures. A third removes the robust normalization, and a fourth adds the time channel.

Why not just…?

Why not just run Min-kNN Distance with embeddings instead of Levenshtein? This is the real threat, because it keeps the black-box setting and fixes the surface-string weakness. We include it as a baseline at matched $N$. The signature approach can win only by using within-trajectory order, which set-of-embeddings distance discards. The token-shuffle and summary-statistic ablations decide this. If embedding Min-kNN ties us, the contribution narrows to calibrated set-level testing, and we will report it that way.

Why not just compare summary statistics like mean entropy and length? It is cheap, and contamination plausibly lowers entropy globally. This is our first ablation. The mechanistic finding of narrowing into a few directions (Xie et al.) suggests that contamination changes how confidence evolves across steps, not only its average. If the unordered-statistics MMD matches the signature MMD, the path machinery is unnecessary. That would be a publishable negative result.

Why not just train LOS-Net on contaminated examples? Auditors rarely have labeled contaminated examples for a new model or benchmark. LOS-Net's positional attention also has no guarantee under format shifts. The cross-format evaluation settles whether that matters in practice. If LOS-Net transfers well, our advantage reduces to being label-free.

Why not just apply MIN-K% to the benchmark set? RLVR does not optimize token likelihood, and Min-kNN Distance reports that likelihood detectors fail there. Our RLVR condition tests this directly.

Proposed Experiments

Controlled contamination. We fine-tune open models on a random half of GSM8K test items and of MSCalc items, following Xie et al.'s leakage protocol. We also RLVR-train on a seen subset, following the Min-kNN protocol. The untouched halves and freshly generated MSCalc items serve as controls.

Pretraining exposure. WikiMIA and BookMIA provide pretraining-exposure cases.

Metrics. We measure per-item AUC, and set-level power as a function of contaminated fraction (5–100%) and set size. We also measure false-positive rate on clean-versus-clean splits, which should sit at the nominal level if the permutation test is valid, and wall-clock cost per item.

Robustness to rewriting. We rewrite the suspect items by paraphrase and by inserting filler reasoning steps. A reparameterization-invariant test should keep its power under step padding. Min-kNN and LOS-Net should lose more.

Decisive checks. The claim holds if two conditions are met together. First, the signature MMD beats both the unordered-statistics MMD and the token-shuffle ablation in power, under RLVR contamination and with $N \le 8$. Second, the test stays calibrated on clean splits. Robust normalization should additionally beat unnormalized signatures on early-terminating traces, as the stopped-process theory predicts. If it does not, that part of the theory is not the operative constraint on realistic trace lengths.

Main confound. A difficulty mismatch between suspect and control items could produce a spurious difference. Procedurally generated MSCalc controls are exactly matched in template and step count, so they bound this confound.