Abstract
As 6G networks evolve, "Generative AI as a Service" (GaaS) is poised to become a core utility. However, the deployment of Large Language Models (LLMs) faces a critical trade-off between the low latency of edge servers and the high-quality output of central cloud models. This paper proposes a training-free framework that utilizes In-Context Learning (ICL) to make intelligent task-offloading decisions. By modeling both communication and LLM-specific inference delays, the system dynamically balances response speed and generation quality based on user preferences. Experimental results demonstrate that this ICL-based approach achieves system rewards comparable to traditional Deep Reinforcement Learning (DRL) methods while offering greater flexibility and reduced computational overhead.
Introduction
The transition toward 6G necessitates a robust infrastructure for Generative AI. Current deployments face a fundamental challenge: small-scale models (e.g., Llama3-8B) at the network edge provide rapid responses but may lack the reasoning depth required for complex tasks. Conversely, massive models (e.g., GPT-4o) in the central cloud offer superior quality but introduce significant network transmission delays.
Traditional offloading strategies often rely on Deep Reinforcement Learning (DRL), which requires extensive training data and complex fine-tuning to adapt to varying network conditions. This work addresses these limitations by introducing a "training-free" decision-making process that leverages the inherent reasoning capabilities of LLMs themselves to manage network resources.
Methodology
The proposed framework consists of two core innovations: a specialized delay model for LLM inference and an ICL-based decision-making engine.
Comprehensive Delay Modeling
Unlike standard offloading models that focus primarily on data transmission, this framework explicitly accounts for the multi-phase nature of LLM inference. The delay is partitioned into the Prompt/Prefill phase (Time-To-First-Token, or TTFT) and the Decode phase (Time-Per-Output-Token, or TPOT).
By accurately modeling these phases, the system can calculate the total time a user waits for a complete response, allowing for more nuanced offloading decisions that consider the specific computational characteristics of different LLM architectures.
ICL-based Decision Making
The system uses an LLM as a meta-optimizer. Instead of a dedicated neural network, a "meta-prompt" is constructed containing a task description and a Prioritized Experience Pool. This pool contains examples of past "good" and "bad" decisions. By processing this context, the LLM learns to choose between "local" (edge) and "offload" (cloud) execution based on real-time network states and task requirements (e.g., a quick chat vs. high-accuracy coding). To prevent stagnation, an Epsilon-Greedy strategy is employed, allowing the system to occasionally explore new offloading patterns and update its experience pool.
Experimental Setup
The framework was evaluated in a simulated 3GPP urban network environment with 20 users. Tasks were split into "regular" (e.g., translation) and "quality-preferred" (e.g., scientific reasoning). Model performance was benchmarked using real-world data from Llama3-8B/70B, GPT-4o, Gemma 7B, and Gemini 1.5 Pro, with quality indices derived from public benchmarks like Chatbot Arena and MMLU.
Key Results and Discussion
The evaluation yielded several critical insights:
- Performance Parity: The ICL-based method achieved system rewards comparable to optimal DRL baselines, proving that LLMs can effectively manage network optimization tasks without explicit training.
- Convergence and Adaptation: The system typically converged to a stable decision-making state within 40-50 iterations. For quality-preferred tasks, it successfully learned to prioritize cloud offloading to meet accuracy requirements, despite the increased latency.
- Latency Optimization: The combination of Gemma 7B at the edge and Gemini 1.5 Pro in the cloud provided the lowest overall service delay, largely due to their superior TTFT performance.
Conclusion
This work demonstrates the viability of using LLMs as intelligent controllers in 6G networks. By treating offloading as a natural language reasoning task, the framework bypasses the heavy training requirements of traditional AI. Future work will focus on addressing the security risks of edge-deployed LLMs and optimizing the token overhead associated with large meta-prompts in the ICL process.
Summary updated