Abstract
This paper introduces a unified reinforcement learning (RL) framework designed to train lightweight, on-device Large Language Models (LLMs) to recognize their own reasoning limits. Unlike traditional approaches that rely on external routers to decide when to offload queries to the cloud, this method enables the on-device model to autonomously invoke cloud assistance. By treating routing and reasoning as a joint optimization problem, the framework significantly narrows the performance gap between local and cloud-only execution while maintaining efficiency and privacy.
The Challenge of On-Device Reasoning
Deploying LLMs involves a persistent trade-off: powerful cloud-based models offer superior reasoning but suffer from high latency, privacy risks, and significant operational costs. Conversely, small on-device models are fast and private but frequently fail on complex logical tasks.
Current hybrid solutions typically employ a router—a separate, smaller classifier that predicts whether a local model can handle a given prompt. However, these routers often rely on surface-level linguistic features rather than the actual logical depth of a problem. Furthermore, because the router and the LLM are trained independently, the system cannot leverage the model's internal "sense" of its own competence.
Proposed Methodology: Unified Routing and Reasoning
The authors propose a methodology that integrates routing decisions directly into the LLM's reasoning process during the post-training (RL) phase.
The framework is built on four key pillars:
- Hierarchical Rewards: The model is trained with a reward system that prioritizes local success but also rewards "judicious" offloading. An accuracy reward is given for correct local answers, while a slightly lower coordination reward is provided if the model correctly identifies its need for help and the cloud model subsequently solves the problem.
- Group-Adaptive Policy Gradient (GAPG): To stabilize training, GAPG uses a group-level gradient estimator to reduce variance by sampling multiple responses per prompt.
- Adaptive Prompt Filtering: To prevent "policy collapse"—where a model learns to always ask for help or never do so—this mechanism ensures training batches maintain a balance of locally solvable and cloud-required prompts.
- Early Exit Mechanism: The model is trained to use a specific tag (e.g.,
<unknown> I need external assistance </unknown>) to halt failing reasoning chains early, saving local compute resources.
Experimental Evaluation
The approach was evaluated on rigorous reasoning benchmarks, including Countdown (arithmetic puzzles) and MATH-lighteval (algebra, geometry, and number theory). Generalization was tested on unseen datasets such as MATH-500, AMC23, and AGI-Eval-Math.
Key Results and Insights
Testing was conducted on LLaMA-3.2 (1B/3B) and Qwen2.5 (1.5B/3B) models, with DeepSeek-R1 serving as the cloud expert.
- Significant Performance Gains: On the Countdown task, the method improved accuracy by approximately 30% compared to local-only tuning, nearly matching cloud-heavy strategies.
- Superior Self-Routing: The "self-routing" LLM consistently outperformed systems using dedicated external routers (like DeBERTa-v3-large), demonstrating that the LLM is the best judge of its own capabilities.
- Token Efficiency: Collaborative models became more efficient by recognizing "dead ends" in their logic and stopping generation sooner than models forced to attempt every problem locally.
- Robust Generalization: The ability to know when to ask for help transferred effectively to unseen datasets, where traditional routers often failed due to changes in problem surface features.
Limitations and Future Directions
The current framework relies on verifiable rewards, making it most suitable for tasks with clear "right" or "wrong" answers, such as mathematics or coding. Extending this to open-ended tasks like creative writing would require more complex reward models, such as "LLM-as-a-Judge." Future work could also explicitly incorporate monetary costs or latency directly into the RL reward function to create economically optimized collaborative systems.
Summary updated