Amortized Inference Search

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 117 papers27 conceptsfrozen 2026-10-04

The performance of generative models is often limited by a "generation-verification gap": models are frequently better at recognizing a correct answer than they are at generating one on the first try. To bridge this gap, researchers often use inference-time search strategies like Best-of-N sampling or Monte Carlo Tree Search (MCTS). While effective, these methods are computationally expensive, requiring hundreds of forward passes to produce a single high-quality output.

A growing body of research focuses on amortizing this search—transferring the benefits of expensive test-time computation directly into the model's static weights. By training on the results of its own successful searches, a model can learn to produce high-reward outputs in a single, efficient forward pass.

The Mechanism of Amortization

Amortization treats inference-time search as a teacher and the base model as a student. Instead of relying on human-labeled data, the system uses its internal capabilities to explore a search space and identify optimal paths.

In the "Sharpening Mechanism" described by Audrey Huang and Cyril Zhang, models improve by fine-tuning on their own high-likelihood, high-quality outputs. This process "sharpens" the probability distribution, shifting mass toward the successful sequences discovered during search. This allows the model to achieve the performance of complex sampling strategies using only simple greedy decoding (Self-Improvement in Language Models: The Sharpening Mechanism).

Learning from Search Trajectories

Beyond just selecting the best final answer, amortization can involve training on the entire "reasoning path" or "denoising trajectory" that led to a successful result.

Search as Variational Inference

Recent work has unified these approaches under the framework of Variational Expectation-Maximization (EM). In this view, the search for high-reward samples is the "E-step," and distilling those samples into the model weights is the "M-step."

Jae-Woo Lee and Minsu Kim apply this to diffusion alignment, showing that gradient-based guidance and importance sampling can be baked into a model to maintain high performance at standard inference speeds (Diffusion Alignment as Variational Expectation-Maximization). Similarly, in the domain of sentence compression, Demian Gholipour Ghalandari uses reinforcement learning to explore the search space during training, making the final model 4,000 times faster than unsupervised methods that search at inference time (Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning).

Efficiency and Scaling

The primary motivation for amortization is efficiency. By moving the computational burden from deployment to training, researchers can achieve high-quality results on resource-constrained hardware.

For example, the GIFT framework by Giorgio Giannone and Faez Ahmed reduces the compute needed for image-to-CAD synthesis by 80% by internalizing successful search paths discovered during an offline bootstrapping phase (GIFT: Bootstrapping Image-to-CAD Program Synthesis via Geometric Feedback). This suggests that "smart" training on search results can effectively compensate for smaller model scales, allowing tiny models to outperform much larger ones on specialized tasks (MCTS-SQL: Light-Weight LLMs can Master the Text-to-SQL through Monte Carlo Tree Search).

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_amortized_inference_search_5005e196,
  title = {Amortized Inference Search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/amortized-inference-search/rdr_5005e196399442a0bff5a8bd9ce43c8a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_amortized_inference_search_5005e196_v1,
  title = {Amortized Inference Search},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/amortized-inference-search/rdr_5005e196399442a0bff5a8bd9ce43c8a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}