Introduction
Generative AI (GAI) models, particularly those based on the Transformer architecture, are increasingly central to modern applications but remain notoriously resource-intensive. As we move toward 6G networks, the challenge lies in balancing the massive data rates promised by next-generation communication with the limited battery and processing power of mobile devices. Researchers have proposed a "cloud-edge-end" collaborative system designed to distribute GAI tasks across different network layers, effectively slashing latency while maintaining high accuracy.
Problem Statement: The Inference Latency Trap
Current inference solutions typically suffer from one of two drawbacks. Cloud-only inference introduces high communication latency, making real-time applications sluggish. Conversely, local or edge-only inference often relies on smaller models that lack the depth and accuracy of their cloud-based counterparts. The primary goal of this research is to minimize "inference latency"—the sum of model computation time and the transmission delays between user devices, edge nodes, and the cloud.
Methodology: Multi-Level Collaborative Framework
The proposed framework treats the network as a hierarchy consisting of Layer 0 (User Device), Layer 1 (Edge), and Layer 2 (Cloud). The system optimizes performance through three core innovations:
- Confidence-Based Probabilistic Offloading: Instead of using a rigid threshold, the system employs temperature-scaled softmax to calculate a confidence score. An S-shaped probability curve determines whether to offload a task. This ensures that even high-confidence results from local models can be verified by larger models when necessary, smoothing transitions between layers.
- Attention-Based Pruning: When offloading to a larger cloud model, the local device "pre-digests" the input. It uses internal attention weights to identify and prune redundant tokens (e.g., prepositions), reducing data transmission volume and speeding up cloud-side computation.
- Optimized Early Exit: To prevent "overthinking" on simple tasks, the system implements an early exit strategy within Transformer models. If two adjacent layers produce nearly identical results (based on a similarity threshold $\tau$), the model terminates early, saving significant computational cycles.
Experimental Setup and Datasets
The system was evaluated using the IMDB dataset for sentiment analysis. Three models from the BERT family were utilized to test the multi-level architecture: bert-base-uncased, bert-large-uncased, and bertweet. Experiments were conducted on a high-performance setup featuring an RTX 4090, with simulated 6G network delays.
Key Insights
- Mitigating the "Overthinking" Tax: Standard architectures force models to complete every layer even when a correct conclusion is reached early. The "Early Exit" mechanism effectively recaptures this wasted time.
- Small Models as Smart Filters: By using local attention mechanisms to prune input for the cloud, the user device acts as a "smart compressor" that understands semantic importance, rather than just a dumb terminal.
- Graceful Degradation: The probabilistic S-curve for offloading avoids the brittleness of hard thresholds, balancing local speed and cloud accuracy more effectively.
Results and Performance
The collaborative system demonstrated significant efficiency gains in a simulated 6G environment:
- Latency Reduction: The system achieved up to a 17% reduction in inference time compared to existing multi-level systems like Tabi.
- Accuracy vs. Speed: With a confidence threshold of 0.8 and an early exit parameter of 0.00001, the system maintained over 90% accuracy.
- Tunable Performance: The shortest recorded inference time was 398.53 ms (at 0.7 confidence and $\tau=0.01$), providing a clear trade-off "knob" for engineers to tune based on specific application needs.
Conclusion and Future Work
While highly effective for discriminative tasks like sentiment analysis, the current pruning methods may face challenges in full-scale generative tasks where every word is vital for context. Future research will focus on making the system more dynamic to handle the second-by-second fluctuations in bandwidth and edge availability characteristic of real-world 6G environments.
Summary updated