Abstract
Pass@k is a standard metric for verifiable tasks like coding and mathematics, measuring whether at least one of $k$ samples is correct. While directly optimizing pass@k during post-training improves multi-sample performance, it frequently degrades pass@1 (single-shot accuracy). This paper characterizes this trade-off as a result of gradient conflict induced by prompt interference. We show that pass@k objectives implicitly reweight training data toward "hard" prompts. When these prompts interfere negatively with the broader dataset, the resulting update direction rotates away from the pass@1 objective, leading to a performance "see-saw."
The Pass@k vs. Pass@1 Trade-off
In the development of reasoning models, researchers often fine-tune LLMs to maximize the probability that at least one generated solution passes a verifier. However, empirical evidence shows a recurring problem: as the model gets better at finding a correct answer in $k$ tries, its first-guess accuracy (pass@1) often suffers. This is problematic for real-world applications where latency and compute costs make single-shot reliability a priority.
Theoretical Framework: Hard-Example Mining
The core of the issue lies in how the pass@k objective treats different prompts. Mathematically, the policy gradient for pass@k implicitly applies a weight to each prompt proportional to $(1 - p)^{k-1}$, where $p$ is the current success probability.
As $k$ increases, this weighting scheme becomes an aggressive "hard-example miner." For large $k$, the model virtually ignores "easy" prompts (where $p$ is high) and focuses almost exclusively on "hard" prompts (where $p$ is low). In some experiments, the authors found weight disparities as high as $10^{28}$, effectively turning a diverse dataset into a highly biased subset of outliers.
Prompt Interference and Gradient Conflict
Focusing on hard examples is not inherently damaging. The degradation occurs when these hard prompts exhibit negative interference with the rest of the dataset. The researchers introduced a "similarity kernel" to measure how an update for one prompt affects the performance of others.
If the features required to solve a "hard" prompt contradict the features needed for the majority of "easy" prompts, the pass@k update will conflict with the pass@1 update. When $k$ is large enough, the influence of these negatively interfering hard prompts dominates, rotating the total gradient until it forms an obtuse angle with the pass@1 direction.
Empirical Results on Reasoning Tasks
The theory was validated using competition-level math problems (MATH dataset) and state-of-the-art models like DeepSeek-R1-Distill-Llama-8B and Qwen-7B.
Key findings include:
- Inner Product Flips: For Llama-8B, the pass@k reweighting caused the gradient alignment to flip from positive to negative, resulting in a strongly negative inner product (-0.613) between the pass@k update and the pass@1 objective.
- Outlier Dominance: In the Qwen-7B model, even though "easy" prompts outnumbered "hard" ones by 21 to 1, the extreme exponential weighting caused the hard, interfering prompts to completely dictate the model's learning trajectory.
Conclusion and Future Directions
This work provides a rigorous explanation for why "training for second chances" can break an LLM's first guess. The identified mechanism—gradient conflict driven by reweighted prompt interference—suggests that future pass@k optimization methods must account for gradient alignment. Potential solutions include "gradient surgery" to mask conflicting updates or exploring alternative objective functions that balance exploration with single-shot reliability.
Summary updated