TL;DR
Researchers have developed Contrastive Beam Search (CBS), a method that allows Large Language Models (LLMs) to "self-distill" at inference time without any additional training or external reward models. By comparing a model’s own internal probabilities when primed for "excellent" versus "poor" reasoning, CBS extracts a steering signal that significantly improves performance on complex math, coding, and science tasks.
What Problem Does This Paper Address?
While LLMs have become remarkably capable, they often stumble on multi-step reasoning. A popular training technique called Self-Distillation Fine-Tuning (SDFT) helps by letting a model act as its own teacher, using expert demonstrations to guide its learning. However, SDFT is computationally expensive, requires gradient updates, and—most importantly—needs ground-truth "expert" demonstrations that aren't available when a user asks a new question at test time.
The authors ask: Can we operationalize the self-distillation principle at inference time, without updating parameters or having access to expert answers?
Methods
The core innovation is the use of counterfactual contexts. Instead of a real expert demonstration, the authors append two fixed textual templates to the user's query:
- Positive Context ($c^+$): e.g., "This is an example for a response with excellent reasoning:"
- Negative Context ($c^-$): e.g., "This is an example for a response with wrong reasoning:"
By calculating the Pointwise Mutual Information (PMI)—specifically the log-odds ratio of a candidate answer under these two conditions—the authors derive an implicit reward signal. If a specific reasoning path is much more likely when the model is told to be "excellent" than when it is told to be "wrong," that path is prioritized.
Mathematically, this leads to a Gibbs reweighting of the model's base distribution. However, the authors prove (Proposition 1) that you cannot simply pick the best token one by one (greedy decoding) because a "good" local token might lead to a "bad" future trajectory. To solve this, they propose Contrastive Beam Search (CBS), which evaluates these rewards over chunks of tokens, allowing the search to account for future trajectory quality.
Datasets
The authors evaluated CBS across three challenging benchmarks:
- MATH500: Multi-step quantitative mathematical reasoning.
- HumanEval: Python code generation tasks scored by unit tests.
- GPQA: Graduate-level science questions in biology, physics, and chemistry.
They tested these on a variety of model scales, including Qwen3.5 (0.8B, 2B), Qwen2.5-7B, and DeepSeek-Math-7B-Instruct.
Key Insights and Ideas
- Semantic Contrast is the Key: The performance boost isn't just from having more text in the prompt. Ablation studies showed that "neutral" prompts (e.g., "control group one" vs "control group two") provided no benefit. The gains come specifically from the semantic opposition between good and bad reasoning.
- The "Future Correction" Term: The paper highlights a theoretical gap in local decoding. To truly follow the optimal self-distilled policy, a decoder needs a "future correction" term that is usually intractable. CBS approximates this by searching in $K$-token blocks.
- Diversity vs. Steering: Unlike other methods like Power Sampling that often require low temperatures (which can reduce output diversity), CBS works well at higher temperatures. It steers the model toward correctness while maintaining the diversity needed to explore different reasoning paths.
Key Results
CBS consistently outperformed standard sampling, low-temperature decoding, and standard beam search across almost all settings.
As shown in the table above, CBS achieved the best or tied-best results in 11 out of 12 experimental settings. Notable gains included a 13.8% absolute improvement over low-temperature sampling on certain tasks.
In terms of efficiency, while CBS requires roughly 1.5x the wall-clock time of standard beam search due to the extra forward passes for the counterfactual contexts, it often produces more concise answers, leading to fewer total completion tokens for some models.
Limitations and Future Work
- Manual Context Design: The counterfactual prompts (like "excellent reasoning") are currently hand-crafted. The authors suggest that learning or searching for task-adaptive context pairs could further boost performance.
- Computational Overhead: The method requires two additional forward passes per beam. While manageable on modern GPUs, the authors point to KV-cache sharing as a potential way to reduce this overhead.
- Scaling to Planning: The authors view this as a step toward "continual learning agents" that can systematically propose and evaluate their own internal states to guide multi-step planning without needing weight updates.
Summary updated