The performance of generative models is often limited by a "generation-verification gap": models are frequently better at recognizing a correct answer than they are at generating one on the first try. To bridge this gap, researchers often use inference-time search strategies like Best-of-N sampling or Monte Carlo Tree Search (MCTS). While effective, these methods are computationally expensive, requiring hundreds of forward passes to produce a single high-quality output.
A growing body of research focuses on amortizing this search—transferring the benefits of expensive test-time computation directly into the model's static weights. By training on the results of its own successful searches, a model can learn to produce high-reward outputs in a single, efficient forward pass.
The Mechanism of Amortization
Amortization treats inference-time search as a teacher and the base model as a student. Instead of relying on human-labeled data, the system uses its internal capabilities to explore a search space and identify optimal paths.
In the "Sharpening Mechanism" described by Audrey Huang and Cyril Zhang, models improve by fine-tuning on their own high-likelihood, high-quality outputs. This process "sharpens" the probability distribution, shifting mass toward the successful sequences discovered during search. This allows the model to achieve the performance of complex sampling strategies using only simple greedy decoding (Self-Improvement in Language Models: The Sharpening Mechanism).
Learning from Search Trajectories
Beyond just selecting the best final answer, amortization can involve training on the entire "reasoning path" or "denoising trajectory" that led to a successful result.
- Reasoning Paths: Tianwei Ni and Allen Nie demonstrate that training on traces from Tree-of-Thought search allows models to internalize complex logic. Their method, Unlikelihood Fine-Tuning (UFT), further improves this by teaching the model to "forget" failed paths, resulting in a 180x speedup over active search (Teaching Large Language Models to Reason through Learning and Forgetting).
- Denoising Trajectories: In diffusion models, Fu-Yun Wang and colleagues show that the cost of searching for optimal image generation paths can be shifted to the training phase. By updating weights to internalize the best-performing trajectories, the model generates high-quality images without the overhead of multi-path sampling (Diffusion-Sharpening: Fine-tuning Diffusion Models with Denoising Trajectory Sharpening).
Search as Variational Inference
Recent work has unified these approaches under the framework of Variational Expectation-Maximization (EM). In this view, the search for high-reward samples is the "E-step," and distilling those samples into the model weights is the "M-step."
Jae-Woo Lee and Minsu Kim apply this to diffusion alignment, showing that gradient-based guidance and importance sampling can be baked into a model to maintain high performance at standard inference speeds (Diffusion Alignment as Variational Expectation-Maximization). Similarly, in the domain of sentence compression, Demian Gholipour Ghalandari uses reinforcement learning to explore the search space during training, making the final model 4,000 times faster than unsupervised methods that search at inference time (Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning).
Efficiency and Scaling
The primary motivation for amortization is efficiency. By moving the computational burden from deployment to training, researchers can achieve high-quality results on resource-constrained hardware.
For example, the GIFT framework by Giorgio Giannone and Faez Ahmed reduces the compute needed for image-to-CAD synthesis by 80% by internalizing successful search paths discovered during an offline bootstrapping phase (GIFT: Bootstrapping Image-to-CAD Program Synthesis via Geometric Feedback). This suggests that "smart" training on search results can effectively compensate for smaller model scales, allowing tiny models to outperform much larger ones on specialized tasks (MCTS-SQL: Light-Weight LLMs can Master the Text-to-SQL through Monte Carlo Tree Search).