Collaborative Inference for Large Models with Task Offloading and Early Exiting

Zuan Xie, Yang Xu, Hongli Xu, Yunming Liao, Zhiyuan YaoarXiv 2024 · OpenReview

TL;DR

This paper introduces DTO-EE, a distributed framework that enables large-scale model inference (like LLMs) across multiple resource-constrained edge nodes by combining model partitioning with an "early exit" mechanism. By jointly optimizing task offloading paths and exit confidence thresholds, the system reduces average response delay by up to 41% while slightly improving inference accuracy compared to traditional methods.

What Problem Does This Paper Address?

Deploying large-scale models (e.g., GPT, LLaMA) at the network edge is essential for low-latency "smart city" applications, but these models are often too massive for a single edge server's memory and compute capacity.

While researchers have tried partitioning models across multiple nodes or using "early exiting" (stopping inference early if the model is confident enough), combining these techniques in a real-world environment is difficult. Edge systems are heterogeneous (nodes have different hardware) and dynamic (task arrival rates fluctuate). A static offloading strategy fails because changing an early exit threshold at one node immediately changes the processing load for all subsequent nodes in the pipeline.

Methods

The authors propose the Distributed Task Offloading and Early Exiting (DTO-EE) algorithm. Key technical components include:

  • Model Partitioning & Pipeline Inference: The model is split into sequential sub-models distributed across edge servers. Tasks flow through these nodes like an assembly line.
  • Joint Optimization: Instead of treating offloading and early exits separately, the authors theoretically analyze their "coupled relationship." They use a utility function to balance the trade-off between response delay and accuracy.
  • Distributed Architecture: To handle dynamic environments, the optimization isn't centralized. Instead, nodes execute two coordinated algorithms:
  • DTO-R (Receiver): Nodes receiving tasks calculate their current load and "gradient information" (how much delay would change if load increased).
  • DTO-O (Offloader): Nodes sending tasks use "repulsive factors" to shift traffic away from overloaded successors toward idle ones.
  • Exterior-Point Method: To ensure no node exceeds its physical compute capacity, the authors transform hard resource constraints into a "penalty function" within their optimization objective.

Datasets

The framework was evaluated using two classic benchmarks adapted for edge sub-tasks:

  1. ImageNet: Used with a ResNet101 model partitioned into 4 sub-models with exit branches on the second and third stages.
  2. Tnews (Chinese news text classification): Used with a Bert-large model partitioned into 5 sub-models with exit branches on the second, third, and fourth stages.

Key Insights and Ideas

  • The "Coupling" Effect: The most important insight is that early exit thresholds act as a "valve" for the rest of the network. Lowering a threshold at an early node reduces the traffic downstream, which can prevent bottlenecks but might hurt accuracy. DTO-EE is the first to optimize this valve and the routing path simultaneously.
  • Reuse-based Evaluation: To avoid the massive overhead of re-evaluating the model every time a threshold changes, the authors use a "one-shot" evaluation method. They record softmax outputs from a validation set once and reuse them to quickly simulate how different threshold settings would impact accuracy and exit rates.
  • Repulsive Factors: The algorithm uses a "repulsive" logic—if a successor node becomes overloaded, the offloader exponentially reduces the probability of sending tasks there, leading to much faster load balancing than traditional "round-robin" or genetic algorithms.

Key Results

The experimental results demonstrate that DTO-EE significantly outperforms baselines like NGTO (Game Theory) and GA (Genetic Algorithm):

  • Latency: Reduced average response delay by 21% to 41%.
  • Accuracy: Improved inference accuracy by 1% to 4% (or 1.1‰ to 5.2‰ in specific BERT configurations) by finding more efficient exit points.
  • Robustness: In dynamic environments where task rates were randomly changed, DTO-EE maintained a stable response delay (standard deviation of ~29ms), whereas baselines fluctuated wildly (standard deviations of 63ms to 84ms).

Limitations and Future Work

  • Architecture Specificity: The current model assumes a linear pipeline of sub-models. Future work could explore more complex Directed Acyclic Graph (DAG) structures where sub-tasks might run in parallel rather than just sequentially.
  • Communication Overhead: While the local communication (RUR/RUS messages) is efficient, the paper does not deeply explore the impact of extreme network congestion on the optimization phase itself.
  • Validation Dependency: The "reuse-based" threshold optimization relies on having a representative validation dataset available at the edge to predict accuracy; if the real-world data distribution shifts significantly (data drift), the thresholds might become sub-optimal.

Summary updated

Abstract

In 5G smart cities, edge computing is employed to provide nearby computing services for end devices, and the large-scale models (e.g., GPT and LLaMA) can be deployed at the network edge to boost the service quality. However, due to the constraints of memory size and computing capacity, it is difficult to run these large-scale models on a single edge node. To meet the resource constraints, a large-scale model can be partitioned into multiple sub-models and deployed across multiple edge nodes. Then tasks are offloaded to the edge nodes for collaborative inference. Additionally, we incorporate the early exit mechanism to further accelerate inference. However, the heterogeneous system and dynamic environment will significantly affect the inference efficiency. To address these challenges, we theoretically analyze the coupled relationship between task offloading strategy and confidence thresholds, and develop a distributed algorithm, termed DTO-EE, based on the coupled relationship and convex optimization. DTO-EE enables each edge node to jointly optimize its offloading strategy and the confidence threshold, so as to achieve a promising trade-off between response delay and inference accuracy. The experimental results show that DTO-EE can reduce the average response delay by 21%-41% and improve the inference accuracy by 1%-4%, compared to the baselines.
BibTeX
@misc{xie2024collaborative,
  author = {Zuan Xie and Yang Xu and Hongli Xu and Yunming Liao and Zhiyuan Yao},
  title = {{Collaborative Inference for Large Models with Task Offloading and Early Exiting}},
  year = {2024},
  eprint = {2412.08284},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.08284},
}

Loading related content...

Figure