TL;DR
Current LLM evaluation metrics fail to account for reasoning stability—how consistently a model can generate correct answers across multiple attempts. This paper introduces G-Pass@$k$, a novel evaluation metric that uses varying correctness thresholds to measure both a model's peak potential and its operational consistency. The authors reveal that even state-of-the-art models experience severe performance degradation when strict reasoning consistency is required, though long-Chain-of-Thought (Long-CoT) models show significantly better robustness.
What Problem Does This Paper Address?
When deploying Large Language Models (LLMs) in the real world, we typically use sampling parameters (like temperature and top-$p$) to diversify responses. If a user gets an incorrect answer, they often hit "regenerate" until they get a satisfactory result.
Traditional evaluation metrics fail to capture this dynamic reliably:
- Greedy Accuracy only evaluates a single, deterministic path.
- Pass@$k$ measures the probability of generating at least one correct answer in $k$ attempts.
The flaw? Pass@$k$ can drastically overestimate a model's practical utility. If a model has a success rate of only 10%, Pass@$k$ might still yield a high score because the model got lucky once out of many tries. In production, an unpredictable model that requires constant regenerations is highly unstable. This paper tackles the challenge of defining and measuring reasoning stability—the probability that a model consistently generates correct solutions across multiple independent attempts.
Methods
The authors propose G-Pass@$k_\tau$ (Generalized Pass@$k$), which introduces a correctness threshold parameter, $\tau \in (0, 1]$.
Instead of asking if a model got at least one answer right, G-Pass@$k_\tau$ calculates the probability that a model generates at least $\lceil \tau \cdot k \rceil$ correct solutions within $k$ attempts.
- When $\tau \to 0$, the metric mathematically relaxes and converges to the standard Pass@$k$ (measuring pure potential).
- When $\tau = 1.0$, the metric becomes ultra-stringent, requiring all $k$ generations to be correct (measuring absolute stability).
To provide a single, unified metric, they introduce mG-Pass@$k$, which calculates the area under the G-Pass@$k_\tau$ curve integrated across thresholds from $\tau = 0.5$ to $1.0$. This captures both high accuracy and high stability.
Datasets
The authors evaluate models across several benchmarks, including a brand-new dataset:
- LiveMathBench (New): A tough, bilingual (English and Chinese) mathematics benchmark curated from the latest 2024/2025 competitions and examinations to ensure zero data contamination. It includes subsets from the China National Mathematical Olympiad (CNMO), China’s College Entrance Examination (CCEE), the American Mathematics Competition (AMC), and the William Lowell Putnam Mathematical Competition (WLPMC).
- MATH500-L5: A subset of the hardest (Level 5) problems from the standard MATH dataset.
- AIME2024-45 & AIME2025: Questions from the American Invitational Mathematics Examination.
Key Insights and Ideas
- The Illusion of Competence via SFT: Supervised Fine-Tuning (SFT) on contaminated or specific datasets artificially inflates a model's Greedy Accuracy. However, under the hood, its stability deteriorates drastically. The model effectively memorizes superficial patterns rather than developing robust reasoning capabilities.
- Scaling Parameters Don't Fix Instability: Simply doubling or tripling parameter size (e.g., moving from Qwen2.5-32B to 72B) yields minimal improvements in stability metrics. The base capabilities remain bottlenecked; true stability requires better training paradigms or test-time scaling rather than raw parameter expansion.
- Long-CoT as a Stabilizer: "o1-like" models (such as QwQ and DeepSeek-R1) exhibit remarkably flat G-Pass curves across varying sampling temperatures and top-$p$ settings. Their internal backtracking and self-reflection mechanisms act as a buffer, allowing them to catch and correct errors introduced by random sampling.
Key Results
The authors ran extensive evaluations across general, mathematical, and o1-like reasoning models.
As the correctness threshold $\tau$ becomes more stringent, performance plummets across the board. Even DeepSeek-R1, the most robust model tested, drops from an 81.1% Greedy Accuracy to 69.5% when absolute consistency ($\tau = 1.0$) is demanded on LiveMathBench. Standard models fare much worse; Qwen2.5-72B-Instruct drops from 51.7% down to 29.0%.
The full breakdown of this stability degradation on LiveMathBench can be seen in the main results table:
Limitations and Future Work
While insightful, the paper notes a few limitations:
- Domain Specificity: The experiments focus entirely on mathematical reasoning. While math is an excellent proxy for logical reasoning, the metric's behavior on code generation or open-ended symbolic logic remains unproven.
- API Constraints: Some closed-source models have incomplete data due to API rate limits, costs, and service instabilities during large-scale sampling runs.
- Language Scope: The benchmarks are strictly limited to English and Chinese.
Future work points toward expanding G-Pass@$k$ to broader reasoning domains and designing training objectives that explicitly incentivize reasoning stability rather than single-shot accuracy.
Summary updated