Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has become the standard for training reasoning LLMs. However, this process often leads to a "Normalization Squeeze," where valid but rare reasoning paths are systematically suppressed. We propose Amortized Reasoning Tree Search (ARTS), a method that decouples generation from verification. By using a novel Flow Matching objective, ARTS enables a frozen base model to achieve performance comparable to fully fine-tuned models while preserving the latent diversity necessary for solving complex, long-tail problems.
The Problem: Normalization Squeeze and Capability Extinction
Standard RL optimization for LLMs is inherently "mode-seeking." Because it relies on finite sampling, correct reasoning paths with low initial likelihood (rare paths) are often missed during training. As the model updates its parameters to boost the probability of sampled correct paths, the normalization requirement forces the probability of unsampled rare paths toward zero. This phenomenon, termed the Normalization Squeeze, creates an "Extinction Zone" for complex reasoning traces.
Amortized Reasoning Tree Search (ARTS)
To address capability extinction, ARTS shifts the focus from parameter-level internalization to inference-time deliberation.
Decoupled Architecture
ARTS maintains a frozen "Proposer" (the base model) to preserve its original entropy and latent knowledge. A separate, lightweight "Verifier" is trained to navigate the proposer's output space.
Flow Matching Objective
Unlike traditional Process Reward Models (PRMs) that use discriminative objectives, ARTS employs a Flow Matching objective grounded in GFlowNet theory. It treats reasoning as a probability flow through a state-space graph, training the verifier to estimate the "conservation of flow." This approach is more robust in high-entropy search spaces where early errors can be subtle.
Sparse Deep Fork (SDF)
Training the verifier involves the Sparse Deep Fork method. Instead of exhaustive tree construction, the model samples a main trajectory and stochastically "forks" at intermediate steps to explore alternative paths, providing the necessary signal for the flow matching objective.
Experimental Results
Evaluated on the MATH-500 benchmark, ARTS demonstrates significant improvements in reasoning robustness.
Key Findings
- Performance Parity: ARTS achieved 74.6% accuracy (BoN@16), effectively matching a fully fine-tuned 7B GRPO policy (74.7%) using only a 1.5B verifier and a frozen base model.
- Rescuing the Long Tail: On the "Extinction Set"—problems where RL-tuned models failed completely—ARTS recovered 6.9% accuracy, a 4x improvement over standard verifiers.
- Domain Robustness: In high-entropy domains like Counting & Probability, ARTS showed a +5.3% gain over pointwise PRMs, highlighting the advantage of global flow-based verification over local step-wise rewards.
Conclusion and Future Work
ARTS demonstrates that disentangling verification from generation is a viable and often superior alternative to standard RL fine-tuning for complex reasoning. While currently limited by the proposer's latent capacity and inference-time costs, future work could involve "Iterative ARTS," using search-discovered paths to fine-tune the base model in a closed loop, potentially overcoming the visibility barrier permanently.
Summary updated