Local-Cloud Inference Offloading for LLMs in Multi-Modal, Multi-Task, Multi-Dialogue Settings

Liangqi Yuan, Dong-Jun Han, Shiqiang Wang, Christopher Brinton

Introduction

Deploying Large Language Models (LLMs) involves a fundamental engineering trade-off. Local execution on edge devices offers low latency, privacy, and zero cost but often relies on smaller, text-only models with limited reasoning capabilities. Conversely, cloud-based models like GPT-4o provide superior multi-modal reasoning at the expense of network latency and per-token usage costs.

Existing offloading strategies typically treat this as a binary decision for isolated queries. However, real-world AI assistants operate in complex "Three-M" environments: Multi-modal (processing text, images, and video), Multi-task (switching between assistant duties, recommendations, and editing), and Multi-dialogue (maintaining context across turns). This paper introduces TMO (Three-M Offloading), a system that intelligently manages these trade-offs.

Overview of the TMO system in a multi-modal kitchen environment, illustrating the choice between local and cloud inference across different camera views.

Methodology: Resource-Constrained Reinforcement Learning

TMO utilizes a Resource-Constrained Reinforcement Learning (RCRL) framework to optimize the offloading policy. The system observes the current state—including dialogue history and task category—to select an action: whether to use a local or cloud LLM and which specific data modalities (e.g., first-person vs. overhead camera views) to transmit.

The RCRL Loop

The policy is trained to maximize a multi-faceted reward function:

  1. Response Quality: Scored by a "Judge LLM."
  2. Task-Modality Association: Calculated using a pre-trained CLIP model to ensure only relevant visual data is uploaded, preventing bandwidth waste.
  3. Resource Penalties: Negative rewards for high latency and usage cost constraints.
The RCRL framework architecture, showing the interaction between the state, action selection (LLM and modality), and the multi-objective reward function.

To handle the non-deterministic nature of LLM responses during offline training, the authors employed a nearest-neighbor strategy to estimate the quality of unseen state-action pairs. Several RL backbones, including A2C, PPO, and DQN, were evaluated with added "safety layers" to strictly enforce resource budgets.

The M4A1 Dataset

To evaluate TMO, the researchers released the M4A1 dataset, comprising 21,014 dialogues. The dataset covers four primary tasks (Assistant, Recommendation, Query, Message Editing) across 2–5 dialogue turns. It includes three distinct image views: first-person, side, and overhead. Responses were generated using a combination of local (Phi-3-mini) and cloud (GPT-4o) models, with GPT-4o providing reference scores.

Key Insights and Results

The RC-A2C (Resource-Constrained A2C) variant of TMO demonstrated superior performance across all metrics:

  • Optimal Balance: It achieved a total reward of 0.85, outperforming both "Local-only" (0.74) and "Cloud-only" (0.75) baselines.
  • Constraint Adherence: TMO maintained zero violations of latency and cost constraints. In contrast, "LLM-as-Agent" baselines—where an LLM decides the offloading strategy—struggled with precise numerical constraints, frequently exceeding budgets.
  • Efficiency Gains: Compared to a cloud-only approach, TMO reduced average latency by approximately 17% and slashed usage costs by nearly 30% while maintaining or improving response quality through selective modality pruning.

Conclusion

TMO demonstrates that intelligent, context-aware offloading is essential for the next generation of multi-modal AI assistants. By treating offloading as a resource-constrained optimization problem rather than a simple binary choice, TMO balances the reasoning power of the cloud with the efficiency of local devices. Future work will explore privacy-centric constraints and multi-server environments involving various cloud tiers.

Summary updated

BibTeX
@inproceedings{yuan2025local,
  author = {Liangqi Yuan and Dong-Jun Han and Shiqiang Wang and Christopher Brinton},
  title = {{Local-Cloud Inference Offloading for LLMs in Multi-Modal, Multi-Task, Multi-Dialogue Settings}},
  booktitle = {MobiHoc},
  year = {2025},
  doi = {10.1145/3704413.3764429},
}

Loading related content...

Figure