Beam Search as Test-Time Self-Distillation via Counterfactual Contexts

Su Ee Tan, Xiaotong Ji, Rasul Tutunov, Haitham Bou-Ammar, Matthieu ZimmerarXiv 2026 · arXiv

TL;DR

Researchers have developed Contrastive Beam Search (CBS), a method that allows Large Language Models (LLMs) to "self-distill" at inference time without any additional training or external reward models. By comparing a model’s own internal probabilities when primed for "excellent" versus "poor" reasoning, CBS extracts a steering signal that significantly improves performance on complex math, coding, and science tasks.

What Problem Does This Paper Address?

While LLMs have become remarkably capable, they often stumble on multi-step reasoning. A popular training technique called Self-Distillation Fine-Tuning (SDFT) helps by letting a model act as its own teacher, using expert demonstrations to guide its learning. However, SDFT is computationally expensive, requires gradient updates, and—most importantly—needs ground-truth "expert" demonstrations that aren't available when a user asks a new question at test time.

The authors ask: Can we operationalize the self-distillation principle at inference time, without updating parameters or having access to expert answers?

Methods

The core innovation is the use of counterfactual contexts. Instead of a real expert demonstration, the authors append two fixed textual templates to the user's query:

  1. Positive Context ($c^+$): e.g., "This is an example for a response with excellent reasoning:"
  2. Negative Context ($c^-$): e.g., "This is an example for a response with wrong reasoning:"

By calculating the Pointwise Mutual Information (PMI)—specifically the log-odds ratio of a candidate answer under these two conditions—the authors derive an implicit reward signal. If a specific reasoning path is much more likely when the model is told to be "excellent" than when it is told to be "wrong," that path is prioritized.

Mathematically, this leads to a Gibbs reweighting of the model's base distribution. However, the authors prove (Proposition 1) that you cannot simply pick the best token one by one (greedy decoding) because a "good" local token might lead to a "bad" future trajectory. To solve this, they propose Contrastive Beam Search (CBS), which evaluates these rewards over chunks of tokens, allowing the search to account for future trajectory quality.

Datasets

The authors evaluated CBS across three challenging benchmarks:

  • MATH500: Multi-step quantitative mathematical reasoning.
  • HumanEval: Python code generation tasks scored by unit tests.
  • GPQA: Graduate-level science questions in biology, physics, and chemistry.

They tested these on a variety of model scales, including Qwen3.5 (0.8B, 2B), Qwen2.5-7B, and DeepSeek-Math-7B-Instruct.

Key Insights and Ideas

  • Semantic Contrast is the Key: The performance boost isn't just from having more text in the prompt. Ablation studies showed that "neutral" prompts (e.g., "control group one" vs "control group two") provided no benefit. The gains come specifically from the semantic opposition between good and bad reasoning.
  • The "Future Correction" Term: The paper highlights a theoretical gap in local decoding. To truly follow the optimal self-distilled policy, a decoder needs a "future correction" term that is usually intractable. CBS approximates this by searching in $K$-token blocks.
  • Diversity vs. Steering: Unlike other methods like Power Sampling that often require low temperatures (which can reduce output diversity), CBS works well at higher temperatures. It steers the model toward correctness while maintaining the diversity needed to explore different reasoning paths.

Key Results

CBS consistently outperformed standard sampling, low-temperature decoding, and standard beam search across almost all settings.

Performance comparison of CBS across 4 models and 3 benchmarks.

As shown in the table above, CBS achieved the best or tied-best results in 11 out of 12 experimental settings. Notable gains included a 13.8% absolute improvement over low-temperature sampling on certain tasks.

In terms of efficiency, while CBS requires roughly 1.5x the wall-clock time of standard beam search due to the extra forward passes for the counterfactual contexts, it often produces more concise answers, leading to fewer total completion tokens for some models.

Figure 1: Inference efficiency of (a) Qwen2.5-7B and (b) DeepSeek-Math-7B-Instruct on the MATH500 benchmark.

Limitations and Future Work

  • Manual Context Design: The counterfactual prompts (like "excellent reasoning") are currently hand-crafted. The authors suggest that learning or searching for task-adaptive context pairs could further boost performance.
  • Computational Overhead: The method requires two additional forward passes per beam. While manageable on modern GPUs, the authors point to KV-cache sharing as a potential way to reduce this overhead.
  • Scaling to Planning: The authors view this as a step toward "continual learning agents" that can systematically propose and evaluate their own internal states to guide multi-step planning without needing weight updates.

Summary updated

Abstract

Self-Distillation Fine-Tuning (SDFT) enables a language model to act as its own teacher: by conditioning on a demonstration, the model produces an implicit reward via pointwise mutual information, which guides on-policy learning without external supervision. However, SDFT operates at training time: it requires gradient updates and access to expert demonstrations, making it inapplicable at inference. We propose test-time self-distillation, a decoding-time method that extracts a steering signal from the self-distillation framework without any parameter updates, reward models, or training data. Our key insight is that counterfactual contexts, i.e. fixed textual templates that hypothetically prime the model for excellent versus poor reasoning, can substitute for the demonstration. The log-odds ratio of a candidate answer under these two counterfactual conditions defines a new reward signal. We derive the optimal KL-regularized policy under this reward, which takes the form of a Gibbs reweighting of the base distribution. Crucially, this reweighting is global: it cannot be decomposed into independent per-token operations without ignoring future trajectory quality. We therefore approximate the target distribution via beam search. Experiments on mathematical reasoning (MATH500), code generation (HumanEval), and graduate-level science QA (GPQA) across multiple model scales show that test-time self-distillation improves over standard sampling, low temperature, beam search and power sampling baselines on average, demonstrating that the self-distillation principle can be operationalized at inference time.
BibTeX
@misc{tan2026beam,
  author = {Su Ee Tan and Xiaotong Ji and Rasul Tutunov and Haitham Bou-Ammar and Matthieu Zimmer},
  title = {{Beam Search as Test-Time Self-Distillation via Counterfactual Contexts}},
  year = {2026},
  eprint = {2609.37041},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.37041},
}
Figure