ORBIT: On-policy Exploration-Exploitation for Controllable Multi-Budget Reasoning

Kun Liang, Clive Bai, Xin Xu, Chenming Tang, Sanwoo Lee, Weijie Liu, Saiyong Yang, Yunfang Wu

TL;DR

This paper introduces ORBIT, a novel framework for training Large Language Models (LLMs) to perform reasoning at different "effort" levels. ORBIT allows LLMs to dynamically adjust their reasoning depth based on the query, achieving a better trade-off between accuracy and computational cost without sacrificing performance.

What Problem Does This Paper Address?

LLMs have shown impressive capabilities in complex reasoning tasks, often by generating lengthy "chains of thought" (CoT). However, applying this deep reasoning uniformly to all queries is computationally expensive and often unnecessary for simpler problems. Existing methods for controlling reasoning effort at inference time struggle with accuracy and reliability, as estimating the optimal reasoning budget is inherently difficult. Furthermore, the trade-off between reasoning cost and accuracy is often fixed during training, limiting flexibility in real-world deployment scenarios. This paper tackles the challenge of creating LLMs that can intelligently adapt their reasoning depth on-the-fly, offering controllable reasoning behaviors across multiple levels of effort.

Methods

ORBIT employs a two-stage approach:

  1. Exploration Phase (Multi-stage RL): This phase uses a novel "Expansion-Compression Loop" to discover specialized reasoning "frontiers" for each desired reasoning mode (e.g., low, mid, high, extra-high effort).
  2. Expansion: The model is initially allowed a large context budget to focus on semantic correctness and establish a global performance ceiling.
  3. Sequential Compression: The context budget is progressively tightened. This introduces truncation, which can destabilize training. ORBIT uses a logarithmic compression schedule to manage this pressure and re-equilibrate the policy at each stage. This process yields per-mode "expert" policies that represent optimal trade-offs along the accuracy-length frontier.
  4. Exploitation Phase (On-Policy Distillation): The per-mode expert policies are then fused into a single, unified student model.
  5. Multi-Teacher On-Policy Distillation (OPD): This technique uses the expert models as teachers to guide the student. OPD is preferred over standard supervised fine-tuning (SFT) to avoid exposure bias, as it uses on-policy rollouts from the student model itself.
  6. Mode-aware Initialization via Model Merging: To combat the "cold-start" problem in OPD, the student model is initialized by merging the parameters of the expert models. This provides a more stable starting point and accelerates convergence.

Datasets

The paper evaluates ORBIT on several mathematical reasoning tasks and multi-task benchmarks:

  • Mathematical Reasoning: AIME 2024, AIME 2025, BeyondAIME.
  • Multi-task Benchmarks: GPQA-Diamond, MMLU-Pro.

For training, the authors use:

  • DAPO-Math-17K dataset (for DeepSeek-Distill-Qwen-1.5B).
  • Polaris-53K dataset (for Qwen3-4B-2507-Thinking and Openmath-Nemotron-7B).

Key Insights and Ideas

  • The "Expansion-Compression Loop" is crucial: This novel strategy effectively navigates the complex trade-off between reasoning depth and computational cost by first exploring the performance ceiling and then systematically tightening constraints to discover efficient policies.
  • On-Policy Distillation is key for consolidation: Using OPD, rather than offline distillation, helps the student model learn to generate its own reasoning trajectories effectively, preventing issues like exposure bias.
  • Mode-aware initialization via model merging significantly improves training stability: This simple yet effective technique provides a strong starting point for the distillation process, leading to faster convergence and better performance.
  • Controllable reasoning is achievable and beneficial: ORBIT demonstrates that LLMs can be trained to exhibit distinct, predictable reasoning behaviors across different modes, offering a practical solution for balancing performance and efficiency.

Key Results

  • ORBIT achieves controllable reasoning behavior across multiple modes, with clear separation and monotonic trade-offs between reasoning effort and task performance.
  • The model demonstrates competitive reasoning density within each mode, often achieving higher accuracy with fewer tokens compared to RL-based efficient reasoning methods.
  • On the AIME 2024 benchmark, ORBIT's Low, Mid, High, and Xhigh modes achieved accuracies of 28.7%, 34.5%, 38.8%, and 42.6% respectively, with corresponding token lengths of 2.7K, 5.6K, 7.7K, and 8.5K tokens.
  • The framework transfers reliably across different domains, from mathematical reasoning to general question answering, with only minimal performance degradation.
  • Joint RL training (training all modes simultaneously) led to "mode collapse," where different reasoning modes converged to similar behaviors, highlighting the importance of ORBIT's sequential exploration and exploitation strategy.

Limitations and Future Work

  • Computational Cost: Long-form RL training can be computationally expensive, especially for larger models, potentially limiting scalability.
  • Sample Efficiency: On-policy RL training might suffer from low sample efficiency when model output entropy is low, which could hinder further compression progress.
  • Verification Signals: The evaluation primarily focused on tasks with structured verification signals. Extending to tasks where correctness is harder to verify remains an open area.
  • Theoretical Guarantees: The paper does not provide theoretical guarantees on the convergence or optimality of the proposed training procedure.
  • Ethical Considerations: The authors acknowledge potential risks associated with controllable reasoning models, such as misuse, propagation of incorrect conclusions, or reinforcement of biases. They emphasize the need for responsible deployment.

Future work could explore more efficient training methods for larger models, investigate tasks with less structured verification, and develop theoretical underpinnings for the framework's convergence.

Summary updated

Abstract

Recent Large Reasoning Models (LRMs) achieve strong performance by leveraging long-form Chain-of-Thought (CoT) reasoning, but uniformly applying overlong reasoning at inference time incurs substantial and often unnecessary computational cost. To address this, prior work explores various strategies to infer an appropriate reasoning budget from the input. However, such approaches are unreliable in the worst case, as estimating the minimal required reasoning effort is fundamentally difficult, and they implicitly fix the trade-off between reasoning cost and accuracy during training, limiting flexibility under varying deployment scenarios. Motivated by these limitations, we propose ORBIT, a controllable multi-budget reasoning framework with well-separated reasoning modes triggered by input. ORBIT employs multi-stage reinforcement learning to discover Pareto-optimal reasoning behaviors at each effort, followed by on-policy distillation to fuse these behaviors into a single unified model. Experiments show that ORBIT achieves (1) controllable reasoning behavior over multiple modes, (2) competitive reasoning density within each mode, and (3) integration of these frontier policies into a single unified student model while preserving clear mode separation and high per-mode performance.
BibTeX
@misc{liang2026orbit,
  author = {Kun Liang and Clive Bai and Xin Xu and Chenming Tang and Sanwoo Lee and Weijie Liu and Saiyong Yang and Yunfang Wu},
  title = {{ORBIT: On-policy Exploration-Exploitation for Controllable Multi-Budget Reasoning}},
  year = {2026},
  eprint = {2601.08310},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.08310},
}
Figure