Distributed Collaborative Inference System in Next-Generation Networks and Communication

Chuan Zhang, Xixi Zheng, Xiaolong Tao, Chenfei Hu, Weiting Zhang, Liehuang Zhu

TL;DR

As we transition toward 6G, the computational weight of Generative AI (GAI) remains a major bottleneck for mobile devices. This paper introduces a distributed "cloud-edge-end" collaborative system that uses probabilistic offloading, attention-based pruning, and optimized early exits to slash inference latency by up to 17% without sacrificing accuracy.

What Problem Does This Paper Address?

Generative AI models, particularly those based on the Transformer architecture, are notoriously resource-hungry. While 6G networks promise ultra-high speeds and low latency, running massive models on a smartphone or an IoT sensor is still impractical.

Current solutions typically fall into two traps:

  1. Cloud-only inference: Offloading everything to the cloud solves the computation problem but introduces significant communication latency, which is unacceptable for time-sensitive applications.
  2. Static offloading: Existing collaborative systems often use "hard" thresholds to decide when to offload tasks, which lacks the flexibility to handle varying network conditions or complex inputs effectively.

The authors aim to bridge this gap by creating a multi-level system that intelligently distributes the workload across the entire network hierarchy.

Methods

The proposed system relies on a three-pronged strategy to optimize the inference pipeline:

1. Multi-Level Deployment & Probabilistic Offloading

The framework distributes models of increasing size across three layers: End devices (Layer 0), Edge nodes (Layer 1), and the Cloud (Layer 2).

Instead of a simple binary decision to offload, the system uses Confidence-based Probabilistic Offloading. It calculates the model's confidence in its output using temperature-scaled softmax. If confidence is low, it offloads. If it's high, it uses an S-shaped probability curve to decide whether to offload anyway, allowing higher-tier models to potentially refine the result.

Inference workflow showing the cloud-edge-end hierarchy

2. Attention-Based Pruning

To minimize communication overhead, the system leverages the "understanding" of the smaller, lower-layer model. It identifies unimportant tokens (like prepositions or redundant words) using attention weights and prunes them before sending the data to the next layer. This reduces the payload size and speeds up the subsequent model's processing time.

3. Optimized Early Exit

To combat "overthinking"—where a deep model continues calculating even after it has reached a correct internal representation—the authors implemented an enhanced early exit mechanism. By comparing the similarity of outputs between adjacent Transformer layers, the model can stop early if the results have stabilized.

Mechanism for early exit within a single model

Datasets

The system was evaluated using the IMDB dataset, a standard benchmark for sentiment analysis. The researchers tested the framework using several BERT-family models:

  • bert-base-uncased (Base size)
  • bert-large-uncased (Large size)
  • bertweet (Trained specifically on Twitter data)

Key Insights and Ideas

  • The S-Curve Advantage: The move from hard thresholds to a probabilistic S-curve for offloading (see Figure 4 in the paper) allows for a much smoother trade-off between speed and accuracy. It prevents the system from being "trapped" by a confident but incorrect result from a small model.
  • Pruning as a Bridge: Using attention weights as a pruning guide is a clever way to make the "wasted" computation of a failed local inference useful for the next stage of the pipeline.
  • Ensemble Finalization: Rather than just taking the cloud's word for it, the system uses a weighted ensemble of results from all layers that processed the task, which helps recover accuracy lost during pruning.
The S-shaped probability curve for offloading decisions

Key Results

The experimental results, conducted on a high-performance setup (RTX 4090, Xeon Gold CPU) simulating real-world network delays, show:

  • Latency Reduction: The system reduced inference time by up to 17% compared to the state-of-the-art Tabi framework.
  • Accuracy Maintenance: In its "Balanced" configuration (Confidence threshold 0.8, Early exit $\tau$ 0.00001), the system maintained an accuracy rate of over 90% while significantly cutting time.
  • Flexibility: The system can be tuned for "Accuracy Priority" (using higher thresholds) or "Time Priority" (yielding results in as little as ~398ms).

Limitations and Future Work

While the results are promising for discriminative tasks (like classification and sentiment analysis), the paper does not extensively discuss how this multi-level pruning and ensemble approach would translate to generative tasks (like long-form text synthesis), where token dependencies are more complex.

The authors note that future work will focus on making the system even more dynamic to adapt to highly variable network environments where bandwidth and congestion fluctuate rapidly.

Summary updated

Abstract

With the rapid advancement of artificial intelligence, generative artificial intelligence (GAI) has taken a leading role in transforming data processing methods. However, the high computational demands of GAI present challenges for devices with limited resources. As we move towards the sixth generation of mobile networks (6G), the higher data rates and improved energy efficiency of 6G create a need for more efficient data processing in GAI. Traditional GAI, however, shows its limitations in meeting these demands. To address these challenges, we introduce a multi-level collaborative inference system designed for next-generation networks and communication. Our proposed system features a deployment strategy that assigns models of varying sizes to devices at different network layers. Then, we design a task offloading strategy to optimise both efficiency and latency. Furthermore, a modified early exit mechanism is implemented to enhance the inference process for single models. Experimental results demonstrate that our system effectively reduces inference latency while maintaining high-quality output. Specifically, compared to existing work, our system can reduce inference time by up to 17% without sacrificing the inference accuracy.
BibTeX
@article{zhang2025distributed,
  author = {Chuan Zhang and Xixi Zheng and Xiaolong Tao and Chenfei Hu and Weiting Zhang and Liehuang Zhu},
  title = {{Distributed Collaborative Inference System in Next-Generation Networks and Communication}},
  journal = {IEEE Trans. Cogn. Commun. Netw.},
  year = {2025},
  doi = {10.1109/tccn.2025.3527679},
}
Figure