Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization

Zishun Yu, Tengyu Xu, Di Jin, Karthik Abinav Sankararaman, Yun He, Wenxuan Zhou, Zhouhao Zeng, Eryk Helenowski, Chen Zhu, Sinong Wang, Hao Ma, Han Fang

TL;DR

Current long-reasoning LLMs are "uni-modal," wasting massive compute on trivial questions. This paper introduces Inference Budget-Constrained Policy Optimization (IBPO), a framework that allows models to adaptively allocate their "thinking time" based on query difficulty, achieving up to 2x the efficiency of standard self-consistency methods on mathematical reasoning tasks.

What Problem Does This Paper Address?

Recent breakthroughs in LLM reasoning, like OpenAI’s o1 and DeepSeek-R1, rely on scaling "test-time compute"—essentially letting the model think longer through extensive Chain-of-Thought (CoT) or self-correction. However, these models often exhibit an inefficient "uni-modal" behavior: they might spend 30 seconds "thinking" about what $1 + 1$ equals, just as they would for a complex calculus problem.

A long reasoning-chain model spent more than enough inference time on a trivial problem.

This lack of difficulty-awareness leads to unnecessarily high inference costs, increased latency, and a larger carbon footprint. The authors aim to bridge this gap by enabling adaptive reasoning, where a model learns to "spend" its inference budget on hard problems while solving easy ones quickly.

Methods

The authors frame the problem as Constrained Reinforcement Learning. Instead of just maximizing accuracy, they maximize a "utility" objective subject to a specific inference budget constraint (e.g., "only use expensive long-reasoning for 25% of queries").

The IBPO Algorithm

The core contribution is Inference Budget-Constrained Policy Optimization (IBPO). The technical workflow is elegant:

  1. Bi-level Optimization: They first solve for an optimal response distribution in a non-parametric (tabular) space, which is a convex program.
  2. Weighted SFT Update: They show that this complex optimization actually simplifies into a very practical weighted Supervised Fine-Tuning (SFT) update. The model is trained on correct responses, but each sample is weighted based on how much it contributes to maximizing the "reward margin" while staying under budget.
  3. Adaptive Sequential Voting (ASV): To test this, they created a response format where the model can choose to output a single quick answer (cheap) or a sequence of trials to reach a consensus (expensive).

The authors also provide a unifying view of existing iterative SFT methods, showing that popular techniques like Rejection Sampling Fine-Tuning (RFT) and Reward-ranked Fine-Tuning (RAFT) are actually special, unconstrained cases of their framework.

SFT methods from an optimization point-of-view.

Datasets

The study focuses on mathematical problem-solving using the following:

  • Training: A curated version of the MATH dataset and the SDPO dataset (a step-wise preference dataset). They constructed specific subsets: $D_{SV}$ (Sequential Voting) and $D_{ASV}$ (Adaptive Sequential Voting) to train the model on different reasoning lengths.
  • Evaluation: The MATH500 benchmark, a 500-sample subset of the MATH testing split.

Key Insights and Ideas

  • Difficulty-Aware Allocation: The model naturally learns a "difficulty-aware" policy. When the budget is tight (e.g., $q_+ = 25\%$), the model allocates almost all its voting budget to Level 4 and 5 problems, while solving Level 1 problems with a single trial.
  • RL is Necessary for Self-Correction: A key observation is that prompting or standard SFT alone struggles to enable efficient self-correction. The authors argue that this behavior "emerges automatically during RL" rather than being manually created.
  • Optimization as Weighting: The realization that constrained RL can be reduced to a simple weighted SFT update means this approach can be integrated into existing LLM training pipelines with negligible computational overhead.

Key Results

The authors evaluated IBPO using a LLaMA 3.1 8B Instruct base.

  • Absolute Gains: Compared to the base model, IBPO achieved a 4.14% absolute improvement on MATH500 with a 2.16x inference budget and a 5.74% improvement with a 4.32x budget.
  • Efficiency: These improvements are approximately 2x better than vanilla self-consistency (Majority Voting) under the same compute budgets.
  • Constraint Following: The models strictly adhered to the assigned budgets during testing, demonstrating that the "budget awareness" is robust even when facing distribution shifts from training to testing.

Limitations and Future Work

  • Domain Specificity: The current work is limited to mathematical reasoning. Extending this to coding or general-purpose dialogue remains an open challenge.
  • Small Training Scale: The RL training set used only 10,000 prompts. The authors suggest that scaling the data could further alleviate distribution shift issues.
  • Broader Applications: The authors propose that this constrained optimization framework could be used for Group Fairness (capping responses from privileged groups) or Mixture of Experts (MoE) load balancing (enforcing minimal activation density for experts).

By treating inference time as a finite resource to be optimized, IBPO offers a promising path toward LLMs that are not just more capable, but significantly more responsible and efficient.

Summary updated

Abstract

Solving mathematics problems has been an intriguing capability of large language models, and many efforts have been made to improve reasoning by extending reasoning length, such as through self-correction and extensive long chain-of-thoughts. While promising in problem-solving, advanced long reasoning chain models exhibit an undesired single-modal behavior, where trivial questions require unnecessarily tedious long chains of thought. In this work, we propose a way to allow models to be aware of inference budgets by formulating it as utility maximization with respect to an inference budget constraint, hence naming our algorithm Inference Budget-Constrained Policy Optimization (IBPO). In a nutshell, models fine-tuned through IBPO learn to ``understand'' the difficulty of queries and allocate inference budgets to harder ones. With different inference budgets, our best models are able to have a $4.14$\% and $5.74$\% absolute improvement ($8.08$\% and $11.2$\% relative improvement) on MATH500 using $2.16$x and $4.32$x inference budgets respectively, relative to LLaMA3.1 8B Instruct. These improvements are approximately $2$x those of self-consistency under the same budgets.
BibTeX
@inproceedings{yu2025think,
  author = {Zishun Yu and Tengyu Xu and Di Jin and Karthik Abinav Sankararaman and Yun He and Wenxuan Zhou and Zhouhao Zeng and Eryk Helenowski and Chen Zhu and Sinong Wang and Hao Ma and Han Fang},
  title = {{Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=0ERw2196o1},
}
Figure