Abstract
Test-time training (TTT) and reinforcement learning (TTRL) have emerged as powerful techniques for adapting Large Language Models (LLMs) to complex reasoning tasks during inference. However, these methods are often computationally prohibitive due to a fixed, high-budget rollout strategy. We present OptPO (Optimal Rollout Allocation for Test-time Policy Optimization), a framework that treats reward estimation as a sequential hypothesis testing problem. By employing a Bayesian sequential stopping strategy, OptPO dynamically allocates computational resources, achieving 30–50% token savings while maintaining or improving model performance across mathematical and scientific reasoning benchmarks.
Introduction
Large Language Models frequently encounter distribution shifts when applied to specialized domains or novel problem types. Test-time optimization addresses this by allowing models to "practice" on specific queries before generating a final response. A common approach involves generating multiple reasoning paths (rollouts), using majority voting to identify the correct answer, and updating the model policy accordingly.
The primary inefficiency in current TTRL pipelines is the use of a fixed rollout budget (e.g., 64 samples) for every query, regardless of difficulty. This "one-size-fits-all" approach wastes significant compute on easy problems where a consensus is reached early, while potentially undersampling harder instances. OptPO solves this by introducing an adaptive sampling mechanism that stops generation as soon as statistical confidence in a consensus answer is achieved.
Methodology: Bayesian Sequential Stopping
OptPO reframes the rollout process as a Bayesian Sequential Probability Ratio Test (SPRT). Instead of pre-committing to a fixed number of samples, the framework follows an iterative loop:
- Adaptive Sampling: Rollouts are generated in small batches.
- Statistical Verification: After each batch, the system calculates the posterior confidence in the current leading answer based on the observed distribution of results.
- Early Exit: If the confidence level exceeds a predefined threshold (e.g., 95%), the sampling process terminates immediately.
- On-Policy Optimization: The collected rollouts are used to update the model’s policy using algorithms such as PPO (Proximal Policy Optimization) or GRPO (Group Relative Policy Optimization).
This approach ensures that computational effort is concentrated where it is most needed, effectively reducing the "compute tax" on simpler queries.
Experimental Results
We evaluated OptPO across four rigorous benchmarks: MATH-500, AIME 2024, AMC, and GPQA (graduate-level science). Testing involved multiple architectures, including Qwen2.5-Math and Llama-3-Instruct.
Efficiency and Token Savings
As shown in the experimental data, OptPO consistently reduces token consumption by 30% to 50% compared to fixed-budget baselines. For instance, on the Llama-3.2-1B-Instruct model using GRPO, OptPO achieved a 78.36% token saving on the AMC benchmark while slightly improving the Mean@16 accuracy.
Accuracy and Convergence
Beyond efficiency, OptPO often improves final accuracy. By stopping once a strong consensus is reached, the model avoids the "noise" of low-quality tail-end rollouts that can occur in fixed-budget sampling. Performance curves (Figure 1) illustrate that OptPO reaches optimal accuracy levels much faster in terms of total rollout time than traditional TTRL.
Discussion and Conclusion
OptPO demonstrates that test-time policy optimization can be made practical for real-world deployment by replacing rigid sampling budgets with statistical stopping criteria. The framework is architecture-agnostic and compatible with various RL and SFT pipelines.
Future work will explore extending OptPO to open-ended tasks where verifiable rewards are not readily available and investigating its application to large-scale "Reasoning Models" that utilize extended chain-of-thought processing. Currently, OptPO stands as a significant step toward making high-performance test-time adaptation computationally sustainable.
Summary updated