When evaluating a generative model—whether it is a large language model solving a math problem or a reinforcement learning policy navigating a robot—we typically judge its performance based on its "first guess." Standard decoding strategies like greedy search or beam search force the model to commit to a single token or action sequence based on local probabilities. However, this approach creates an artificial bottleneck: the model may possess the latent capability to generate the correct answer, but standard decoding fails to find it.
To overcome this, a powerful paradigm has emerged: test-time compute scaling via candidate selection (often implemented as Best-of-$N$ sampling, rejection sampling, or inference-time reranking). Instead of relying on a single forward pass, the system generates a diverse pool of $N$ candidate responses or actions and uses a secondary evaluation model—a critic, value function, or reward model—to select the best one. This shifts the burden of accuracy from generation to verification, allowing models to achieve significantly higher performance simply by spending more computational budget at inference time.
The Intuitive Asymmetry: Why Verification Beats Generation
Why does this two-stage "generate-then-select" process work so well? The core intuition lies in the fundamental asymmetry between producing a complex, flawless sequence and recognizing one.
Consider a difficult math problem or a humorous caption for a cartoon. Generating a perfect solution on the first try requires a flawless chain of logic where a single early mistake can derail the entire output. However, evaluating a set of completed solutions is often a much simpler task. As shown by Jiayi Chen et al. in the context of humor generation, models are frequently better at recognizing high-quality content than generating it from scratch. By sampling multiple independent paths, the stochastic nature of the generator explores different regions of the solution space, and the verifier acts as a filter to catch logical errors or generic responses.
In fact, Mathieu Ravaut et al. demonstrated that selecting the best candidate from a pre-generated pool can yield performance gains that exceed years of progress in base model architecture. This strongly implies that our current generative models are not necessarily limited by their internal knowledge, but rather by the standard decoding mechanisms we use to extract that knowledge.
The Mechanics of Test-Time Scaling
The candidate selection framework generally operates via a simple, decoupled architecture consisting of two main components:
- The Proposer (Generator): A base model samples multiple independent candidate outputs (e.g., text responses, action sequences, or reasoning paths) using a non-zero temperature to ensure diversity.
- The Verifier (Scorer): A secondary model evaluates each candidate and assigns a score based on a specific objective (e.g., correctness, safety, or expected reward).
This process can be applied at different granularities. For instance, in structured tasks like Text-to-SQL, Mingqian He et al. showed that generating multiple reasoning paths and using a reward-based verifier allows models to recover from individual logical errors, dramatically boosting execution accuracy. Similarly, Ramya Keerthy Thatikonda et al. leveraged Outcome Reward Models (ORMs) to scale test-time compute over a large set of candidate reasoning paths, trading increased generation cost for higher accuracy in complex deductive tasks.
Furthermore, this selection mechanism can act as a lightweight alignment tool. Instead of undergoing expensive reinforcement learning from human feedback (RLHF) to update base model weights, Shijia Huang et al. demonstrated that preference-guided rejection sampling can filter out untruthful or unsafe responses at inference time, providing a computationally efficient alternative to full model fine-tuning.
Application to Reinforcement Learning and Control
While candidate selection has revolutionized language modeling, its roots and some of its most profound impacts are found in reinforcement learning (RL) and robotics. In continuous control and offline RL, agents face the "curse of extrapolation error"—if an agent selects an action outside the training data distribution, the value function might hallucinate a massive reward, leading to catastrophic failure.
Candidate selection elegantly solves this by constraining the action space. In a "guide-then-select" paradigm, an agent samples multiple candidate actions directly from a learned behavior policy or a generative model (such as a diffusion model) and then executes the one with the highest predicted Q-value. For example, Liyuan Mao et al. showed that generating candidates via a guided diffusion process ensures that all options remain within the safe data distribution, while a learned Q-function picks the optimal move. This drastically reduces the risk of choosing actions based on hallucinated rewards.
This strategy also allows for stable policy improvement without complex gradient updates. In Reinforcement Learning with Action Chunking, the authors optimized policies by sampling action sequences from a generative model and selecting the highest-value chunk, bypassing the need for explicit policy gradients through complex architectures. Similarly, Mitsuhiko Nakamoto et al. introduced Value-Guided Policy Steering (V-GPS), demonstrating that re-ranking action proposals from a frozen robotic foundation model using an external value function improves real-world success rates by over 80% without any fine-tuning.
Step-Level and Multi-Agent Extensions
As the field progresses, candidate selection is moving from a single holistic choice at the end of generation to iterative, step-level guidance. For example, Zhiyuan Hu et al. developed a process reward approach for GUI navigation agents, evaluating and selecting the best action at every single step rather than only at the end of a trajectory. Kele Xu similarly proposed a plug-and-play step-level Q-value model that guides LLM agents through complex tasks by evaluating multiple candidate actions at each decision point, doubling performance without altering the core agent weights.
Even in multi-agent or complex environments, this deliberative process shines. In an exploration of model capabilities, Adrian de Wynter utilized a multi-agent decision-making approach where a central agent selects the most appropriate move from multiple candidate actions generated by expert model calls, improving tactical performance in dynamic environments like the game Doom.
Conclusion
The shift toward test-time compute scaling via candidate selection represents a fundamental realization: generation is hard, but selection is manageable. By decoupling the proposal of ideas from their evaluation, researchers can bypass the instabilities of direct fine-tuning, mitigate extrapolation errors in RL, and unlock the latent reasoning capabilities already present within generative models. As inference hardware scales, the ability to trade compute time for accuracy will remain a cornerstone of robust, reliable AI systems.