Abstract
Device-cloud collaboration holds promise for deploying large language models (LLMs), leveraging lightweight on-device models for efficiency while relying on powerful cloud models for superior reasoning. A central challenge in this setting is determining, for each incoming query, whether it should be processed locally or offloaded to the cloud. Existing approaches typically rely on external routers, which often struggle to determine difficulty from the prompt itself, especially for tasks involving complex reasoning. Motivated by this limitation, we propose enabling on-device LLMs to decide internally whether to invoke cloud assistance at inference time, with this capability instilled through reinforcement learning based post-training. Casting on-device LLM post-training as a reward maximization problem, we design hierarchical rewards to encourage local problem solving and judicious cloud offloading. To solve the resulting problem, we develop an algorithm featuring a group-level policy gradient that stabilizes optimization, together with adaptive prompt filtering that provides complementary learning signals to mitigate policy collapse (i.e., exclusive local execution or exclusive cloud offloading). Extensive experiments on on-device-scale LLaMA and Qwen models across multiple reasoning benchmarks show that our method consistently outperforms baselines and significantly narrows the gap to full cloud LLMs.
Introduction
Deploying Large Language Models (LLMs) on-device (e.g., smartphones, laptops) offers significant advantages in terms of privacy, latency, and offline availability. However, these lightweight models often struggle with complex reasoning tasks that their larger, cloud-based counterparts handle with ease. A common solution is a "device-cloud" hybrid system, where a local model handles simple queries and offloads difficult ones to the cloud.
The primary challenge lies in the routing mechanism. Traditional systems rely on external routers—separate classifiers that predict task difficulty based on the prompt. These routers often fail because they lack insight into the specific local model's internal state and reasoning process, guessing difficulty from surface-level features rather than actual performance. Furthermore, training the router and the model in isolation prevents the model from learning how to collaborate effectively. This paper proposes a unified reinforcement learning (RL) framework that trains on-device LLMs to autonomously decide when to invoke cloud assistance, effectively making the model its own router.
Methodology
The core of the proposed approach is the Group-Adaptive Policy Gradient (GAPG) algorithm. This method treats "calling for help" as a specific action within the model's reasoning trace.
- Unified RL Formulation: The on-device LLM is trained to generate a special
<unknown>token when it determines a task exceeds its capability. This token triggers a call to a powerful "Cloud LLM" (e.g., DeepSeek-R1). - Hierarchical Rewards: To prevent the model from becoming overly reliant on the cloud, a tiered reward system is used. A correct answer generated locally receives a higher reward ($\alpha_a$) than a correct answer obtained via cloud assistance ($\alpha_c$). This hierarchy incentivizes the model to maximize its own reasoning potential.
- GAPG Algorithm: Building on Group Relative Policy Optimization (GRPO), GAPG utilizes group-level averaging to stabilize the optimization process and reduce the high variance common in RL training.
- Adaptive Prompt Filtering: To prevent "policy collapse"—where a model either always offloads or never offloads—the authors implement a filtering mechanism. This ensures training batches contain a balanced mix of prompts solvable locally and those requiring cloud intervention.
The researchers evaluated this methodology using several challenging reasoning benchmarks, including Countdown (symbolic reasoning), MATH-lighteval, and generalization benchmarks like MATH-500, AMC23, MinervaMath, and AGI-Eval-Math.
Key Results and Insights
The experimental results demonstrate that the proposed collaborative approach significantly outperforms traditional routing methods and local-only execution.
- Performance Gains: On the Countdown task, the GAPG-trained model reached nearly 80% accuracy, compared to ~50% for models tuned only for local tasks. Across various MATH benchmarks, the collaborative approach narrowed the gap to full Cloud LLMs significantly. For instance, a Qwen-1.5B model using this method achieved 66.7% accuracy, surpassing traditional router-based baselines (60.8%) and naive offloading (58.2%).
- Self-Awareness via Post-Training: A critical insight is that routing is itself a reasoning task. A model that begins solving a problem and realizes it is stuck (an "early exit") is a far more efficient router than an external system guessing from the prompt alone.
- Token Efficiency: Collaborative models become more token-efficient. Instead of generating long, incorrect reasoning chains for difficult problems, the model learns to stop early and call the cloud, saving local compute resources.
- Budget Robustness: The method remains effective even under strict cloud usage constraints (e.g., 20-40% offloading limit), learning to prioritize cloud calls for the most challenging queries.
Conclusion
This paper introduces a novel framework for device-cloud collaboration by instilling "self-awareness" in on-device LLMs through RL-based post-training. While the current framework relies on verifiable rewards (e.g., math answers), it sets a strong foundation for future work. Future research could extend this to open-ended tasks using "LLM-as-a-Judge" reward models and explore dynamic cloud budgets that adapt to real-time device conditions like battery life and network strength.
Summary updated