Amortized Reasoning Tree Search: Decoupling Proposal and Decision in Large Language Models

Zesheng Hong, Jiadong Yu, Hui PanarXiv 2026 · arXiv

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become the standard for training reasoning LLMs. However, this process often leads to a "Normalization Squeeze," where valid but rare reasoning paths are systematically suppressed. We propose Amortized Reasoning Tree Search (ARTS), a method that decouples generation from verification. By using a novel Flow Matching objective, ARTS enables a frozen base model to achieve performance comparable to fully fine-tuned models while preserving the latent diversity necessary for solving complex, long-tail problems.

The Problem: Normalization Squeeze and Capability Extinction

Standard RL optimization for LLMs is inherently "mode-seeking." Because it relies on finite sampling, correct reasoning paths with low initial likelihood (rare paths) are often missed during training. As the model updates its parameters to boost the probability of sampled correct paths, the normalization requirement forces the probability of unsampled rare paths toward zero. This phenomenon, termed the Normalization Squeeze, creates an "Extinction Zone" for complex reasoning traces.

Figure 1: (a) Conceptual illustration of the Normalization Squeeze where RL suppresses diversity, creating an Extinction Zone for rare paths. (b) Empirical evidence showing the decline in relative log-likelihood for rare and endangered reasoning paths during training.

Amortized Reasoning Tree Search (ARTS)

To address capability extinction, ARTS shifts the focus from parameter-level internalization to inference-time deliberation.

Decoupled Architecture

ARTS maintains a frozen "Proposer" (the base model) to preserve its original entropy and latent knowledge. A separate, lightweight "Verifier" is trained to navigate the proposer's output space.

Flow Matching Objective

Unlike traditional Process Reward Models (PRMs) that use discriminative objectives, ARTS employs a Flow Matching objective grounded in GFlowNet theory. It treats reasoning as a probability flow through a state-space graph, training the verifier to estimate the "conservation of flow." This approach is more robust in high-entropy search spaces where early errors can be subtle.

Sparse Deep Fork (SDF)

Training the verifier involves the Sparse Deep Fork method. Instead of exhaustive tree construction, the model samples a main trajectory and stochastically "forks" at intermediate steps to explore alternative paths, providing the necessary signal for the flow matching objective.

Experimental Results

Evaluated on the MATH-500 benchmark, ARTS demonstrates significant improvements in reasoning robustness.

Table 1: Performance comparison on MATH-500. ARTS (BoN@16) matches the performance of a fully fine-tuned GRPO model while significantly outperforming standard reward modeling approaches in closing the performance gap.

Key Findings

  • Performance Parity: ARTS achieved 74.6% accuracy (BoN@16), effectively matching a fully fine-tuned 7B GRPO policy (74.7%) using only a 1.5B verifier and a frozen base model.
  • Rescuing the Long Tail: On the "Extinction Set"—problems where RL-tuned models failed completely—ARTS recovered 6.9% accuracy, a 4x improvement over standard verifiers.
  • Domain Robustness: In high-entropy domains like Counting & Probability, ARTS showed a +5.3% gain over pointwise PRMs, highlighting the advantage of global flow-based verification over local step-wise rewards.

Conclusion and Future Work

ARTS demonstrates that disentangling verification from generation is a viable and often superior alternative to standard RL fine-tuning for complex reasoning. While currently limited by the proposer's latent capacity and inference-time costs, future work could involve "Iterative ARTS," using search-discovered paths to fine-tune the base model in a closed loop, potentially overcoming the visibility barrier permanently.

Summary updated

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has established itself as the dominant paradigm for instilling rigorous reasoning capabilities in Large Language Models. While effective at amplifying dominant behaviors, we identify a critical pathology in this alignment process: the systematic suppression of valid but rare (low-likelihood under the base model distribution) reasoning paths. We theoretically characterize this phenomenon as a "Normalization Squeeze," where the interplay between mode-seeking policy gradients and finite sampling acts as a high-pass likelihood filter, driving the probability of rare correct traces to statistical extinction. To counteract this collapse without discarding the base model's latent diversity, we propose Amortized Reasoning Tree Search (ARTS). Unlike standard approaches that force internalization via parameter updates, ARTS prioritizes deliberation by decoupling generation from verification. We introduce a Flow Matching objective that repurposes the verifier to estimate the conservation of probability flow, enabling robust navigation through sparse, high-entropy search spaces where traditional discriminative objectives fail. Extensive experiments on the MATH-500 benchmark demonstrate that ARTS achieves a performance of 74.6% (BoN@16), effectively matching fully fine-tuned policies (74.7%) without modifying the generative backbone. Crucially, on the long-tail subset where coupled RL optimization collapses to 0% pass@k, ARTS uniquely recovers significant performance, suggesting that disentangling verification from generation offers a more robust pathway for solving complex reasoning tasks.
BibTeX
@misc{hong2026amortized,
  author = {Zesheng Hong and Jiadong Yu and Hui Pan},
  title = {{Amortized Reasoning Tree Search: Decoupling Proposal and Decision in Large Language Models}},
  year = {2026},
  eprint = {2602.12846},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.12846},
}
Figure