Generative AI as a Service in 6G Edge-Cloud: Generation Task Offloading by In-context Learning

Hao Zhou, Chengming Hu, Dun Yuan, Ye Yuan, Di Wu, Xue Liu, Zhu Han, Charlie Zhang

TL;DR

This paper proposes an edge-cloud collaborative framework for deploying Large Language Models (LLMs) as a service in future 6G networks. By leveraging an innovative in-context learning (ICL) approach enhanced with reinforcement learning principles, the system dynamically optimizes task offloading decisions between the network edge and the central cloud to minimize service delay while guaranteeing content quality—completely bypassing the need for traditional, computationally expensive machine learning training.

What Problem Does This Paper Address?

Generative AI (GAI) and LLMs are poised to become central features of 6G networks, powering everything from semantic communication to network automation. However, deploying these massive models within wireless networks presents a major challenge. Ultra-large models like GPT-4 cannot run on resource-constrained edge servers (like base stations), yet routing every user request to a distant central cloud introduces severe backhaul latency.

Furthermore, mobile users have highly diverse requirements: a casual chatting application demands low latency but can tolerate lower accuracy, whereas a coding or scientific reasoning task prioritizes high-quality output over immediate response times. This paper addresses the challenge of generation task offloading: how to intelligently route diverse user prompts to either a small-scale LLM at the network edge or a large-scale LLM in the cloud to minimize total delay while satisfying individual task quality constraints.

Fig. 1. GAI model deployment and services in wireless networks.

Methods

The authors tackle this optimization problem by first establishing realistic mathematical models for both communication and computation:

  1. Communication Model: Quantifies downlink transmission delay based on link capacity (optimized via a classic proportional fairness algorithm) and factors in backhaul delay for cloud-offloaded tasks.
  2. LLM Inference Model: Rather than treating LLM generation as a black box, the authors split inference delay into two distinct phases: Time-To-First-Token (TTFT) (prefill phase) and Time-Per-Output-Token (TPOT) (decoding phase). This allows them to linearly estimate generation time based on expected token length.
Fig. 2. LLM inference process illustration. (EOS: end-of-sequence).

Instead of using traditional Deep Reinforcement Learning (DRL) to solve the offloading problem—which requires extensive training and fine-tuning—the authors introduce an In-Context Learning (ICL) framework. They use an LLM itself as the decision-maker by feeding it a carefully designed meta-prompt consisting of:

  • Task Descriptions and Rules: Instructions defining the network environment and constraints.
  • Prioritized Experience Replay: A dynamic pool of successful past offloading examples ("good decisions") that achieve high rewards.
  • Exploration Strategy: An $\epsilon$-greedy policy that occasionally forces random offloading choices to discover better routing strategies and continuously update the example pool.
Fig. 3. LLM-enabled in-context learning for task offloading.

Datasets

The paper does not utilize traditional training datasets. Instead, it leverages performance metrics from established LLM benchmarks (such as Chatbot Arena and Multi-task Language Understanding [MMLU]) to define the quality indices ($\tau$) of various models.

For evaluation, the authors simulate a wireless environment based on 3GPP urban networks featuring 20 randomly distributed users requesting a mix of regular tasks (chat, translation) and quality-preferred tasks (coding, scientific reasoning).

Key Insights and Ideas

  • LLMs as Zero-Training Network Optimizers: The core "aha moment" is that an LLM can act as a real-time optimization engine purely through natural language prompting. This eliminates the complex training pipelines, data collection bottlenecks, and rigid architectures of conventional machine learning techniques.
  • Marrying ICL with Reinforcement Learning: By incorporating an $\epsilon$-greedy exploration strategy and an experience replay pool directly into the prompt context, the authors allow the LLM's prompt to "evolve" and converge toward optimal behavior, mimicking a traditional Q-table but using natural language.
  • Tractable Latency Modeling: Breaking down LLM latency into TTFT and TPOT provides a clean, practical framework for network engineers to integrate AI computation times directly into standard wireless resource allocation equations.

Key Results

Through extensive simulations, the authors demonstrated several key findings:

  • DQN-Equivalent Performance: The proposed ICL method (using Llama3-8B or GPT-4o as the decision-maker) achieved system rewards comparable to an optimal conventional Deep Q-Network (DQN) baseline after a brief exploration phase.
  • Constraint Satisfaction: The framework successfully maintained a 100% success rate for quality-preferred tasks during the exploitation phase, ensuring complex tasks were correctly routed to the cloud.
  • Model Combinations Matter: When testing different edge-cloud pairings, the combination of Gemma 7B (edge) + Gemini 1.5 Pro (cloud) achieved the lowest overall average service latency. This is attributed to their exceptionally low TTFT values (e.g., 1/155 second for Gemma 7B) compared to alternatives like Llama2.
Fig. 5. Simulation results and comparisons

Limitations and Future Work

While the framework proves highly effective at reducing latency and maintaining service quality, the authors point out a critical trade-off: deploying LLMs across a distributed edge-cloud architecture inherently increases the network's attack surface. Distributed models are more vulnerable to security threats such as inference and data extraction attacks. Consequently, the authors note that future research will focus on developing secure and robust deployment strategies to safeguard user data and model integrity in 6G environments.

Summary updated

Abstract

Generative artificial intelligence (GAI) is a promising technique towards 6G networks, and generative foundation models such as large language models (LLMs) have attracted considerable interest from academia and telecom industry. This work considers a novel edge-cloud deployment of foundation models in 6G networks. Specifically, it aims to minimize the service delay of foundation models by radio resource allocation and task offloading, i.e., offloading diverse content generation tasks to proper LLMs at the network edge or cloud. In particular, we first introduce the communication system model, i.e., allocating radio resources and calculating link capacity to support generated content transmission, and then we present the LLM inference model to calculate the delay of content generation. After that, we propose a novel in-context learning method to optimize the task offloading decisions. It utilizes LLM's inference capabilities, and avoids the difficulty of dedicated model training or fine-tuning as in conventional machine learning algorithms. Finally, the simulations demonstrate that the proposed edge-cloud deployment and in-context learning task offloading method can achieve satisfactory generation service quality without dedicated model training or fine-tuning.
BibTeX
@article{zhou2025generative,
  author = {Hao Zhou and Chengming Hu and Dun Yuan and Ye Yuan and Di Wu and Xue Liu and Zhu Han and Charlie Zhang},
  title = {{Generative AI as a Service in 6G Edge-Cloud: Generation Task Offloading by In-context Learning}},
  journal = {IEEE Wireless Communications Letters},
  year = {2025},
  url = {https://openreview.net/forum?id=JGUyIniy9P},
}
Figure