TL;DR
BudgetThinker gives Large Language Models (LLMs) an internal clock by periodically injecting lightweight control tokens during generation to notify them of their remaining token budget. Paired with a two-stage Supervised Fine-Tuning (SFT) and curriculum Reinforcement Learning (RL) pipeline, it enables models to dynamically adapt their chain-of-thought reasoning length, delivering state-of-the-art accuracy under strict computation constraints across both text and multimodal benchmarks.
What Problem Does This Paper Address?
Modern reasoning LLMs (such as DeepSeek-R1 and OpenAI o1) excel by "thinking longer" through extended test-time computation. However, uncontrolled Chain-of-Thought (CoT) generation brings high latency and computational cost, making these models ill-suited for real-time systems (e.g., autonomous driving, robotics) or latency-sensitive agents.
Merely requesting brevity in prompts fails: models cannot keep track of how many tokens they have generated and often wander off or run out of tokens mid-thought. Meanwhile, hard truncation cuts off answers before completion, and prior length-control methods either rely on verbose multi-token textual prompts or struggle to maintain accuracy when forced to compress their reasoning.
Methods
BudgetThinker introduces dynamic token insertion alongside a tailored training pipeline:
- Ratio-Based Control Token Injection: Instead of inserting text hints or fixed-step markers, the framework inserts a small, fixed set of $K$ special tokens (e.g., $K=8$) whenever output generation reaches a fractional increment of the budget $B$ (at $t = k \cdot \lfloor B/K \rfloor$). This notifies the model of its consumed budget fraction (e.g., 25%, 50%, 75%) regardless of whether $B$ is 500 or 10,000 tokens.
- SFT with Budget Allocation & Compression: Training data blends original long CoT, direct non-thinking responses, and iteratively compressed CoT traces generated by an LLM compressor. Target budgets are assigned by rounding true response lengths up to multiples of a granularity constant ($T=50$), teaching the model to conclude naturally before exhausting the limit.
- Asymmetric Length-Aware RL (GRPO): Group Relative Policy Optimization optimizes policy behavior with a compound reward combining correctness, formatting, and an asymmetric quadratic length penalty that heavily penalizes exceeding the budget while gently penalizing coming in under budget.
- Curriculum & Mixed Budget Training: RL proceeds through a curriculum that gradually reduces allowed budgets ($B_1 > B_2 > \dots > B_n$), concluded by a randomized multi-budget phase to mitigate catastrophic forgetting.
Prompt: "Please answer within B tokens..."
Generation: <think> ... [Remain 7/8] ... [Remain 6/8] ... </think> **Final Answer** ...
Datasets
The authors evaluate BudgetThinker on both text-based LLMs and Multimodal LLMs (MLLMs):
- Training Datasets:
- LLMs: 41k SFT instances compiled from s1k, LIMO, Bespoke-Stratos-17k, NuminaMath, and MATH (with iterative compressions via GPT-4o-mini). RL used 14k math problems from PRM800K and Eurus-2-RL-Data.
- MLLMs: 111k SFT instances constructed by multi-round GPT-4o compression on 37k VQA CoT examples from RoboBrain-2.0; RL used 2k VQA instances.
- Evaluation Benchmarks:
- Math: MATH-500 and AMC 2023.
- Multimodal Reasoning: ERQA (Embodied Reasoning QA) and SAT (Spatial Aptitude Training).
Key Insights and Ideas
- LLMs need active pacing, not just initial instructions: An initial prompt prompt constraint is forgotten during long autoregressive rollouts. Dynamic in-flight markers act like a countdown timer, allowing the model to adapt its reasoning trajectory on the fly.
- Ratio-based beats fixed-interval signaling: Signaling proportional progress (e.g., $1/8, 2/8, \dots$) is more intuitive for the model than absolute intervals (every 50 or 250 tokens), and it keeps vocabulary overhead constant across any budget size.
- Shorter budgets drive structural compression, not just truncation: Qualitative inspections show that at large budgets (6,000 tokens), the model uses exhaustive branching, generating functions, and multi-step verification. Under tight budgets (500 tokens), it shifts to direct, linear, and deterministic algebraic paths without redundant reflection loops.
Key Results
- Accuracy Gains Across Budgets: On MATH-500 and AMC 2023, BudgetThinker improves accuracy by an average of 4.9% across budgets over baselines, outperforming methods like ThinkPrune, Laser, and Chain of Draft (CoD).
- Small-Budget Resilience: On MATH-500 (1.5B model) with a tiny budget of 500 tokens, BudgetThinker reaches 59.4% accuracy—substantially higher than ThinkPrune (44.0%), Laser (48.4%), and the base model (20.4%).
- Superior Budget Adherence: BudgetThinker achieves close to a 95–98% budget-following ratio at moderate-to-large budgets, drastically reducing the 300-token length–budget discrepancy common in baseline RL.
- Low Overhead: Injects only 14 tokens total overhead (for $K=8$) compared to 50–960 tokens consumed by verbose textual prompting methods like ThoughtTerminator.
Limitations and Future Work
- Inference Runtime Integration: Because control tokens must be injected during generation at exact token intervals, it requires low-level support from serving engines (e.g., custom modifications in vLLM).
- Curriculum Forgetting Dynamics: Training sequentially on smaller budgets initially degrades performance on larger budgets, necessitating an extra mixed-budget stage to recover capacity.
- Domain Scope: Experiments primarily focus on mathematical reasoning and spatial/embodied VQA; further validation is needed for general-domain multi-turn dialogue, open-ended coding, and agentic web-browsing tasks.
Summary updated