Runner-Up Solutions: Exposing Memorized Benchmark Answers by Counting Distinct Correct Paths

When a language model is graded on whether any of its $k$ sampled answers is right (pass@k), or on the best answer a verifier picks from $N$ tries (best-of-N), the score depends only on the single best draw. A single best draw is cheap to get from things that are not reasoning: an answer the model memorized from leaked test data, an answer key retrieved from the web mid-inference, or a lucky guess. Extreme-value theory gives a precise version of this intuition. For the largest of $n$ random draws to clear a high bar, one draw has to clear it. For the second-largest to clear it, two independent draws have to, so its probability shrinks quadratically rather than linearly.

We propose evaluating models by their runner-up correct solutions instead of only their best one. For each benchmark item we sample many solutions, group the correct ones into clusters that share the same solution path, and track how fast the number of distinct correct paths grows with the sampling budget. Our hypothesis is that a genuine capability shows several correct paths. A contaminated item shows one dominant path with almost no runner-up, even when its pass@1 is high. The resulting test needs one model, no training data, and no clean reference models. We test it where contamination is known to exist: models with data leaked during reinforcement-learning post-training, pretraining leaks of MATH-500 in Qwen2.5, and search agents that retrieve answer keys.

Core Claim

Outcome-only multi-sample metrics measure the first order statistic, the maximum over draws. The first order statistic is the quantity most easily inflated by a single exploitable mode. Tight Asymptotics of Extreme Order Statistics proves that for finite-mean distributions the maximum of $n$ draws can grow almost linearly, while every fixed lower rank $\ell \ge 2$ obeys a strict speed limit. The mechanism is the extra exceedance the runner-up requires. Its simulations also show that the ratio of first to second maximum is stable in about 99% of draws, with extreme gaps coming from rare events.

We do not claim those theorems transfer literally. Correctness scores of language models are bounded, often binary, and i.i.d. given the prompt. For i.i.d. binary outcomes, every "ℓ-of-n successes" statistic is a function of the per-item success rate $p$ alone, so it adds nothing to pass@k. What does transfer is the logic of requiring a second independent exceedance. The independence that matters is not between samples but between solution paths. Counting distinct correct paths is the correct analogue of the runner-up order statistic, and it carries information that $p$ does not.

Concretely, suppose the correct probability mass for an item is split across solution paths with masses $q_1 \ge q_2 \ge \dots$. The expected number of distinct correct paths seen in $n$ samples is

$$\mathbb{E}[D_n] = \sum_j \left(1 - (1-q_j)^n\right),$$

where $D_n$ counts distinct correct clusters among $n$ draws. Pass@n is the probability that $D_n \ge 1$. We define distinct-pass$_\ell$@n as the probability that $D_n \ge \ell$. The first is governed by total mass $\sum_j q_j$. The second needs mass on a runner-up path, and appears only after roughly $1/q_2$ samples. A model that has memorized one canonical solution, or that RL has collapsed onto one rewarded trajectory, has large $q_1$ and $q_2 \approx 0$. Its pass curve saturates immediately while its distinct-pass$_2$ curve stays flat. The runner-up ratio $q_1/q_2$, estimated from cluster counts, is our diagnostic, and the extreme-value literature supplies its intuition.

Where This Sits

Three results motivate the proposal. Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination shows Qwen2.5 reaches 65.8% exact match on MATH-500 from an 80% prompt prefix. It also shows that outcome-reward RL with random or inverted rewards "improves" the model on that benchmark but not on the clean RandomCalculation set. That paper studies greedy or single-policy behavior and never looks at the spread of sampled solutions. Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models shows RL on leaked items causes policy collapse that likelihood detectors miss (AUC 0.42–0.53). It catches the collapse by comparing a greedy answer to one answer elicited by an "alternative reasoning" prompt. We test the same collapse passively, through the sampling distribution, and resolve it at the level of correct paths rather than token entropy. Search-Time Data Contamination shows search agents copy answer keys from HuggingFace mirrors, reaching 37.5% versus 17.4% accuracy on contaminated versus clean Humanity's Last Exam items. Copying a key is exactly a single dominant path.

Two lines of work constrain the design. ConStat detects contamination by comparing accuracy on original versus rephrased items against a difficulty curve fitted to about 20 reference models. It fails if the reference cohort is itself contaminated, which is why we aim for a single-model test, though we use ConStat as a baseline. Actions Speak Louder than Words catalogues ways trace-similarity measures fool themselves, including length bias, empty-trace scores of 1.0, and a roughly 56% chance-agreement floor. Any cluster count we report must be normalized against a chance floor and against item-level path multiplicity. Work on pass@k as a training objective, such as Pass@K Policy Optimization and Why Pass@k Optimization Can Degrade Pass@1, treats pass@k as a loss and does not ask what a high pass@k is made of. We treat it as a measurement and decompose it.

Research Program

The program has three parts.

Theory. Using the mixture-of-paths model above, we derive how $D_n$ and distinct-pass$_\ell$@n grow under three regimes: broad reasoning (many paths of comparable mass), collapse (one path), and rare-hit exploitation (tiny $q_1$, which is the guessing or occasional-retrieval case). For the third regime, pass@n climbs like $nq_1$ while distinct-pass$_2$ climbs like $n^2 q_1 q_2$. This is the bounded-variable version of the quadratic penalty in Tight Asymptotics of Extreme Order Statistics. We also give a Good-Turing-style estimator of runner-up mass from singleton and doubleton cluster counts, so the diagnostic can be computed at moderate $n$.

Clustering. The method depends on grouping correct solutions by path, so we build and stress-test that step. For math, one clusterer canonicalizes the sequence of intermediate quantities and equations. A second uses a judge model asked whether two correct solutions take the same approach. We report agreement between them and require that conclusions hold under both. For search agents, the path is the sequence of retrieved sources and tool calls. A trace whose answer comes from a page hosting the benchmark is its own path type.

Calibration. Some clean items admit only one natural path, for example a two-step arithmetic problem. A raw low $D_n$ is therefore not evidence of contamination. We calibrate per item using either the same item's freshly generated twin, built from DyVal-style templates (DyVal) or from rephrasing, or one clean reference model on the same item. The diagnostic is the excess of $q_1/q_2$ over that calibration, normalized by a shuffled-cluster chance floor.

Baselines

Naive reference: pass@k and the per-item success rate. We compute the unbiased pass@k estimator and the empirical success rate from the same $n$ samples, and use either as a contamination score. This is the do-nothing reference. Our theory says it cannot separate a collapsed item from a broad one at equal $p$, so beating it shows that path information matters.

Likelihood detectors: perplexity and Min-K%. These are run as in Detecting Data Contamination from Reinforcement Learning Post-training. On RL-MIA they score 0.42–0.53 AUC. They are included because they remain the default in practice.

Strongest competitor for RL contamination: Self-Critique. The method compares greedy-response entropy with the entropy of a response elicited by an alternative-path prompt. It reports average AUC 0.70 on Qwen2.5-7B-Instruct and 0.64 on DeepSeek-Math-7B-Instruct on the RL-MIA benchmark (AIME 2024/2025, Knights & Knaves, SAT, with 50% injected into RL training). It targets the same phenomenon we do. Self-Critique uses about two generations per item while we use $n$ between 8 and 64, so a naive comparison would be unfair. We therefore also run a budget-matched Self-Critique that averages over $n/2$ critique prompts and temperatures. Beating Self-Critique means exceeding its budget-matched AUC on the same models and splits, not just exceeding 0.70.

Strongest competitor for pretraining contamination: ConStat. ConStat reports detecting 98.4% of sample-specific contamination in controlled settings, and gaps of $\delta \approx 8.3\%$ for Mistral-7b-v0.1 on GSM8k and $\delta \approx 6.6\%$ for Llama-3-70b on ARC. It works at the benchmark level using reference models. We run it with its reference cohort on the same models. We count it as beaten if we match its benchmark-level verdicts with one model, and we also run it in a regime where we deliberately contaminate the reference cohort.

Key ablation. Sample-level answer entropy and self-consistency computed on the same $n$ samples, with no path clustering and no restriction to correct samples. This isolates our key ingredient, counting distinct correct paths rather than overall diversity. If the ablation matches the full method, the order-statistic framing contributes nothing beyond "sample and measure diversity," and we will say so.

Why not just…?

Why not just use pass@k with more samples? For i.i.d. binary grading, pass@k at every $k$ is determined by the per-item success rate. A memorized item with $p=0.9$ and a genuinely easy item with $p=0.9$ produce identical curves. The naive baseline above tests this directly. Our prediction is that its AUC on RL-MIA stays near the likelihood detectors, because RL contamination raises $p$ on both leaked and some clean items.

Why not just measure output entropy over the samples? This is the real threat. Policy collapse lowers entropy, and Self-Critique already exploits that. Our argument is that overall entropy is dominated by the bulk of wrong, idiosyncratic samples. A model can be collapsed on its correct path while still producing diverse wrong answers, or diverse surface wording around one solution. The runner-up statistic looks only at the top of the correct mass, the part that pass@k rewards. The key ablation settles this. If entropy over $n$ samples does as well as distinct-correct-path counting, we will report the order-statistic decomposition as an explanation of why sampling-based detectors work, not as a better detector.

Why not just evaluate on a fresh clean benchmark like LiveMathBench or RandomCalculation? Fresh benchmarks are how we obtain ground truth, but they do not audit the benchmarks people already report. Dynamic generation is largely limited to formal domains, as DyVal acknowledges. Search-Time Data Contamination shows new benchmarks leak through mirrors once released. A per-item, single-model test is complementary, and we use fresh sets as its calibration twins.

Why not just use ConStat? ConStat is strong at the benchmark level, but it needs a clean reference cohort and it flags benchmarks, not items. If ConStat with a clean cohort matches us on every verdict, our advantage reduces to the contaminated-cohort regime, and the experiment below tests exactly that.

Proposed Experiments

RL contamination with known labels. On RL-MIA we reproduce the leaked versus unleaked split for Qwen2.5-7B-Instruct and DeepSeek-Math-7B-Instruct. We sample $n \in \{8, 16, 32, 64\}$ solutions per item at temperature 0.7–1.0 and report AUC versus $n$ for all methods. The decisive check is whether runner-up excess beats budget-matched Self-Critique and the entropy ablation at the same $n$. We will also report the $n$ at which it crosses the reported 0.70.

Pretraining contamination. We compare Qwen2.5-Math-7B and a Llama model on MATH-500 against LiveMathBench and RandomCalculation. We expect Qwen2.5 to show a high runner-up ratio on MATH-500 relative to its own LiveMathBench and RandomCalculation ratios, and Llama to show none. Item-level labels come from partial-prompt completion, following Reasoning or Memorization?: an item counts as leaked if an 80% prefix reproduces the answer. We then repeat the random-reward RL run from that paper. The prediction is that spurious-reward "gains" on MATH-500 come entirely from collapsed items, while distinct-pass$_2$ does not improve. This is the first direct test of whether outcome-reward gains are runner-up-free.

Controlled injection and contaminated references. We fine-tune a 7B model on half of GSM8k test items, either verbatim or paraphrased, so labels are exact. We then rerun ConStat with a reference cohort in which we contaminate an increasing fraction of models. The decisive check is that our single-model test keeps its AUC while ConStat's benchmark verdict degrades.

Search-time contamination. On SimpleQA and a subset of Humanity's Last Exam, we run a search agent $n$ times per item with and without HuggingFace blocked. Blocking defines the ground truth. We predict that contaminated items show one dominant path type (retrieval of a benchmark mirror) and a vanishing runner-up. This would let an auditor flag such items from traces alone, without a blocklist.

What counts as falsification. We discard the core claim if either of two things happens. First, the entropy ablation matches the full method across these settings. Second, calibrated runner-up excess fails to separate leaked from clean items once item path multiplicity is controlled. The second would mean collapsed solution paths are a property of easy items rather than of memorization.