Joint Continual Learning of Local Language Models and Cloud Offloading Decisions with Budget Constraints

Evan Chen, Wenzhi Fang, Shiqiang Wang, Christopher BrintonarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces DA-GRPO, a novel framework for enabling small language models (SLMs) on edge devices to intelligently leverage powerful cloud-based large language models (LLMs) during continual learning. DA-GRPO learns to balance task performance with cloud usage constraints, leading to improved accuracy, reduced forgetting, and stable cloud resource management, which is crucial for real-world deployments.

What Problem Does This Paper Address?

As LLMs become more ubiquitous, there's a growing need to deploy smaller, more efficient models (SLMs) on resource-constrained devices like smartphones and laptops. These SLMs can handle many tasks but struggle with complex reasoning and are prone to "catastrophic forgetting" – losing previously learned skills when adapting to new tasks. To overcome these limitations, SLMs can selectively offload difficult queries to powerful cloud LLMs. However, managing this cloud assistance during continual learning is tricky. Naive approaches often lead to unstable offloading behavior and worsen forgetting. Furthermore, cloud usage incurs costs, so it must be regulated within a budget. This paper addresses the challenge of jointly learning task competence and optimal cloud offloading strategies for SLMs in a continual learning setting, all while respecting a strict cloud usage budget.

Methods

The core contribution is DA-GRPO (Dual-Advantage Group Relative Policy Optimization). Unlike previous methods that treat cloud offloading as a separate routing problem or use fixed reward shaping, DA-GRPO integrates cloud usage constraints directly into the policy optimization process. It extends Group Relative Policy Optimization (GRPO) by introducing a "dual-advantage" mechanism. This mechanism separates the reward for task correctness from the cost of cloud usage, allowing them to be optimized independently but interactively. A dual variable is used to dynamically adjust the influence of the cloud usage constraint on the learning signal. This allows the SLM to learn when to request cloud help as a natural part of its learning process, rather than relying on external rules or pre-defined rewards. The method is designed to be stable and adaptable, even when task distributions change.

Datasets

The paper uses a combination of existing datasets for training and evaluation:

  • MATH-lighteval, ARC-Easy, ARC-Challenge: Used for math and science reasoning tasks.
  • TACO-verified: A dataset of validated code generations.
  • MMLU (Massive Multitask Language Understanding): Used for general knowledge tasks.

The experiments are set up in a continual learning scenario where the model transitions between these task groups.

Key Insights and Ideas

  • Constraint-Aware Learning is Key: Directly incorporating cloud usage as a constraint into the learning objective, rather than as a simple reward penalty, leads to more stable and effective collaboration.
  • Dual Advantage for Balanced Optimization: Separating task reward and cloud cost into distinct advantages allows the model to learn a nuanced trade-off, rather than a monolithic objective.
  • Adaptive Collaboration: The dual variable naturally adapts the collaboration strategy over time, increasing cloud usage when the local model is weak and decreasing it as competence grows, which is crucial for continual learning.
  • Avoiding External Routers: DA-GRPO eliminates the need for separate routing models or classifiers, simplifying the overall system architecture.

Key Results

Experiments on mathematical reasoning and code generation benchmarks demonstrate that DA-GRPO:

  • Improves Post-Switch Accuracy: Achieves higher accuracy on tasks after switching from a previous task, indicating reduced catastrophic forgetting.
  • Substantially Reduces Forgetting: Quantitatively shows a lower forgetting rate compared to baseline methods.
  • Maintains Stable Cloud Usage: Consistently adheres to the prescribed cloud assistance budget.
  • Outperforms Baselines: DA-GRPO shows significant gains over prior collaborative and routing-based approaches, including GRPO, GAPG, and GVPO. For instance, on the MATH-lighteval dataset, DA-GRPO achieved a joint accuracy of 75.5% after switching tasks, compared to 48.5% for a strong baseline.

Limitations and Future Work

  • Limitations: The paper acknowledges that while DA-GRPO provides a principled abstraction, directly optimizing strict constraints can be challenging due to the discrete nature of routing decisions. The current work focuses on post-training adaptation.
  • Future Work: The authors suggest several promising directions:
  • Enabling flexible, user-adjustable collaboration budgets at inference time without retraining.
  • Incorporating auxiliary local signals (e.g., reasoning traces) to further improve cloud inference efficiency and quality.

Summary updated

Abstract

Locally deployed Small Language Models (SLMs) must continually support diverse tasks under strict memory and computation constraints, making selective reliance on cloud Large Language Models (LLMs) unavoidable. Regulating cloud assistance during continual learning is challenging, as naive reward-based reinforcement learning often yields unstable offloading behavior and exacerbates catastrophic forgetting as task distributions shift. We propose DA-GRPO, a dual-advantage extension of Group Relative Policy Optimization that incorporates cloud-usage constraints directly into advantage computation, avoiding fixed reward shaping and external routing models. This design enables the local model to jointly learn task competence and collaboration behavior, allowing cloud requests to emerge naturally during post-training while respecting a prescribed assistance budget. Experiments on mathematical reasoning and code generation benchmarks show that DA-GRPO improves post-switch accuracy, substantially reduces forgetting, and maintains stable cloud usage compared to prior collaborative and routing-based approaches.
BibTeX
@misc{chen2026joint,
  author = {Evan Chen and Wenzhi Fang and Shiqiang Wang and Christopher Brinton},
  title = {{Joint Continual Learning of Local Language Models and Cloud Offloading Decisions with Budget Constraints}},
  year = {2026},
  eprint = {2602.00166},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.00166},
}
Figure