Twin-Family Cluster Validation for Reference-Free Contamination Audits

When a language model has seen a benchmark question during training, its high score may reflect recall rather than skill. Current detectors that work without access to the training data all lean on some outside anchor. One kind uses a panel of clean reference models. Another uses the exact base checkpoint. A third uses a calibrated profile of known-clean questions. Separately, detectors that read the model's output probabilities stop working once the model has been tuned with reinforcement learning. We propose building the anchor at audit time, from the benchmark itself. For each benchmark question we generate a small family of "twins": copies with the same solution structure and surface form but freshly sampled numbers or entities. Because the twins did not exist before the audit, they cannot have been trained on.

The detector is then a question from cluster validation. Cluster validation is the field that scores how well points group together without ground-truth labels, using indices such as the Silhouette coefficient, which compares a point's distance to its own group with its distance to the nearest other group. The question we ask is whether the original question sits inside its twin family in the model's hidden-state space, or stands apart from it. If the model has not memorized the item, the original should be indistinguishable from its twins. If it has, the original becomes an outlier within its own family. Across many contaminated items these outliers should also push in a shared direction. Both properties can be tested exactly by swapping the original with one of its twins, so the test needs no reference model and no labels.

Core Claim

Benchmark contamination at the level of individual items is a violation of exchangeability. Exchangeability here means that, if the model has not memorized a question, the original and its freshly generated twins are statistically interchangeable in how the model represents them. Cluster-validation indices computed within twin families give per-item and set-level tests of this property, and the tests are calibrated by permutation. We claim these tests detect memorization that output-likelihood detectors miss after reinforcement-learning post-training. We also claim that the family structure is what makes them work. Naively applying Silhouette or the Dense Representation Structure Estimator to raw benchmark activations confuses "these questions share a topic" with "these questions were memorized."

Relation to Prior Work

Three papers motivate the gap. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that training with Group Relative Policy Optimization (GRPO), a reinforcement-learning method, shrinks the loss gap between seen and unseen questions. As a result, ten output detectors fall from 89.1% to roughly 50–58% AUROC while pass@1 stays inflated by 8.8–11.7%. That paper never tests hidden-state geometry. ConStat treats contamination as a failure to generalize from originals to rephrased benchmarks and detects 98.4% of sample-specific contamination in controlled settings. It works only at the level of accuracy, and it needs a panel of clean reference models, which its own authors note collapses if the whole ecosystem is contaminated. The survey documents why matching the training data against benchmarks is usually impossible for third-party evaluators.

Three representation-level detectors come closest to this proposal, and each constrains its design. LaRA measures how individual hidden states move under paraphrasing and under blanking of key values. It standardizes those movements against a clean reference set, and it reaches ROC-AUC 0.79 across RL checkpoints. We share its view that hidden-state geometry survives RL. We differ in two ways: our reference is generated per item and is clean by construction, and we test multi-sample cluster structure rather than single-point displacement. RADAR finds that recall and reasoning occupy separable regions of internal feature space. It exploits this with a supervised ensemble trained on labeled examples, whereas we use no labels. CANARY reaches AUROC 1.000 at 1% fine-tuning poisoning by differencing hidden states against the exact base model. Evaluators of benchmarks rarely have that base model. We use CANARY's setting only as an upper reference where a base model happens to exist.

The method comes from the cluster-validation literature. Deep Subspace Clustering with Data Augmentation uses mean Silhouette to select augmentations without ground truth. Zero-shot Clustering of Embeddings shows that Silhouette tracks adjusted mutual information on unseen data, though only after UMAP reduction to 50 dimensions to counter distance concentration. Exploring Structural Degradation in Dense Representations combines a separation-to-radius ratio with effective dimensionality, reaching Kendall's τ≈0.57 with downstream performance. Systematic Analysis of Cluster Similarity Indices warns that such indices carry geometric biases. That warning is why we never interpret raw index values and use only rank-based tests calibrated by permutation.

Research Program

Twin generation. For templatable reasoning benchmarks, twins are minimal edits that keep everything except the values fixed. For GSM8k and AIME-style items, we resample numbers from distributions matched to the originals and verify the new answers by executing the solution program. DyVal items are re-instantiated from their own generating graphs. For ARC-Challenge, a stretch setting, twins come from swapping distractors and permuting answer options. Each family holds one original and $m$ twins, with $m$ between 8 and 32.

Representations. We take the mean-pooled prompt hidden state at every layer and reduce it to 50 dimensions, following the zero-shot clustering pipeline.

Statistics. Write $h(x)$ for this representation. The per-item statistic is the Silhouette value of the original within its family. The "other cluster" for this computation is the nearest other family, so the score adjusts for how spread out each family naturally is. Let $R_i$ be the rank of the original's Silhouette among all $m+1$ family members. Under the no-memorization null, $R_i$ is uniform on $\{1,\dots,m+1\}$. This yields an exact per-item p-value and, by summing ranks, a test over the whole set.

The second statistic targets the shared direction. Define the residual

$$r_i = h(x_i^0) - \frac{1}{m}\sum_{j=1}^{m} h(x_i^j),$$

where $x_i^0$ is the original and $x_i^j$ are its twins. Then measure alignment as

$$A = \frac{\lambda_{\max}\left(\sum_i r_i r_i^\top\right)}{\operatorname{tr}\left(\sum_i r_i r_i^\top\right)},$$

where $\lambda_{\max}$ is the largest eigenvalue and $\operatorname{tr}$ the trace. Memorized items should share a "recall" direction and inflate $A$. The null distribution of $A$ comes from replacing each original with a randomly chosen twin. We choose the layer without labels by taking the maximum statistic over layers and comparing it to the permutation distribution of that maximum.

Calibration. Exchangeability fails if twins differ stylistically from originals. We therefore check calibration on models we train ourselves without leaking the benchmark, using an identical pipeline. This calibrates the twin generator, not the audited model. If the false-positive rate on these clean models exceeds the nominal level, we fix the generator rather than the threshold.

Baselines

Naive reference points. The do-nothing reference is chance, AUROC 0.5. The behavioral reference-free baseline compares accuracy on originals with accuracy on twins inside the same families. This is effectively ConStat with the reference-model panel replaced by our twins. It uses the identical twin sets and model queries, so any gain we show comes from reading representations rather than from the twins themselves. To count as a win, our method must reach a given detection power with fewer items, or detect memorized items that the model also answers correctly on their twins.

Output-likelihood detectors. Min-K%, Loss and LiRA are run exactly as in the Fragility paper, on its Stage I protocol (contaminated supervised fine-tuning, then clean GRPO) and its Stage II protocol (contamination with step-by-step reasoning traces). The paper's reported figure, 89.1% AUROC falling to about 50–58% after GRPO, is the bar. Beating it means staying well above 0.6 AUROC after GRPO, where these detectors sit at chance.

LaRA. LaRA is the strongest existing representation detector. It reports ROC-AUC 0.79 and TPR 31–38% at 5% FPR, combined with Self-Critique, on 60 samples where clean items come from AIME 2026 and members from PRIME-RL. We run it on that same split with its clean reference profile, and on our controlled models with an equal hidden-state extraction budget. Our method gets no clean profile at all. Matching LaRA without a reference is the minimum claim. Exceeding 0.79 AUROC or 38% TPR at 5% FPR on that split is the strong claim.

ConStat. ConStat is the strongest black-box set-level method. It is run with its reference panel, and its reported 98.4% controlled detection rate sets the set-level ceiling we compare against. For real-model agreement we use its reported gaps, δ≈8.3% for Mistral-7b-v0.1 on GSM8k and δ≈6.6% for Llama-3-70b on ARC.

Base-model reference. CANARY is run only where the base model exists, as an upper reference for detectors that see the base checkpoint.

Ablations. These isolate the key ingredient, the family structure. First, the "obvious" detector: Silhouette and the Dense Representation Structure Estimator computed on raw benchmark activations with k-means pseudo-labels, with no twins. Second, plain distance from each original to its twin centroid, with no normalization by the nearest other family. Third, twins replaced by paraphrases, which is closer to LaRA. Fourth, varying the number of twins $m$.

Why not just…?

Why not just compute Silhouette or the Dense Representation Structure Estimator on benchmark activations directly? Benchmark questions cluster by topic whether or not they were memorized, so a tight cluster tells you nothing on its own. Our ablation compares clean and contaminated versions of the same model. We expect this detector's AUROC to sit near chance once topics are balanced. If it does not, the family machinery is unnecessary and we will report that.

Why not just compare accuracy on originals versus twins? This is the real threat, because it uses the same twins and needs no hidden states. Its weaknesses are that it yields one coarse number per item and cannot see memorization when the model also solves the twin. Power curves as a function of the number of audited items will settle the question. If the accuracy gap matches our power at every set size, the representation statistics add nothing for set-level audits, and the contribution narrows to per-item ranking.

Why not just use LaRA or a probe trained on synthetic contamination? Both need clean examples drawn from the audited model's own distribution. That is exactly what is missing when an entire model generation has seen the benchmark, which is the failure case ConStat identifies. Our head-to-head comparison on LaRA's split shows the cost of dropping that requirement.

Why not just use perplexity-based membership inference? The Fragility paper shows these detectors fall to chance after a single epoch of clean GRPO. We reproduce that collapse as a positive control.

Proposed Experiments

We run controlled contamination on Qwen2.5-0.5B, Llama-3.2-1B, SmolLM2-1.7B, Gemma-2-2B and DeepSeek-R1-Distill-Llama-8B. A random half of GSM8k, DyVal and RL-MIA Math items is leaked at 1, 4 or 16 exposures. Each model then optionally receives clean GRPO, following the Fragility protocol, or is contaminated with reasoning traces instead. Per-item metrics are AUROC and TPR at 5% FPR. Set-level metrics are power at α=0.05 as a function of set size, and correlation of each statistic with the measured inflation, defined as accuracy on originals minus accuracy on twins.

Three checks are decisive. First, false positives on our never-leaked models must stay at nominal levels, or the exchangeability premise fails. Second, the statistics must remain well above chance after GRPO, where output detectors collapse. Third, on public models, the set-level verdicts should agree in direction with ConStat's reported gaps.

We expect one honest negative result. Contamination with reasoning traces may lift the twins as well, because the Fragility paper reports confidence rising on unseen but similar tasks. If so, our test will correctly report no item-specific memorization even though family-level skill was acquired from leaked traces. We will measure and state this boundary, not hide it.