Early-Exit Meets Model-Distributed Inference at Edge Networks

Marco Colocrese, Erdem Koyuncu, Hulya Seferoglu

TL;DR

This paper introduces MDI-Exit, a novel framework that combines model-distributed inference (MDI) with early-exit mechanisms for edge networks. By adaptively deciding when to exit a DNN early and how to distribute model layers across edge devices, MDI-Exit significantly improves data processing rates while maintaining accuracy, making AI inference more efficient on resource-constrained edge devices.

What Problem Does This Paper Address?

Edge devices, like smartphones or IoT sensors, often have limited computational power, memory, and battery life, making it challenging to run complex Deep Neural Networks (DNNs) for tasks like image classification. Traditional distributed inference methods either send all data to workers with full models (data-distributed inference, DDI), leading to high communication costs, or split DNNs across workers (model-distributed inference, MDI), which can be complex to manage. Furthermore, not all data inputs require the full processing power of a DNN; some can be classified with high confidence using only a subset of the network's layers. This "network overthinking" leads to wasted computation. The paper addresses the challenge of efficiently performing DNN inference on edge networks by leveraging both model distribution and the concept of early-exit, aiming to reduce latency and increase throughput without sacrificing accuracy.

Methods

The core contribution is the MDI-Exit framework, which integrates model-distributed inference with an early-exit strategy. Here's a breakdown of its key components:

  • Model Partitioning: The DNN is split into segments, with each segment assigned to a different edge worker.
  • Early-Exit Mechanism: At specific "exit points" within the DNN, a classifier provides a confidence score for the current prediction. If this confidence exceeds a predefined threshold, the inference for that data point is terminated early, and the result is sent back.
  • Adaptive Policies: MDI-Exit dynamically determines:
  • Model Allocation: Which worker processes which set of layers.
  • Early-Exit Decisions: When to exit a model early based on confidence scores.
  • Offloading Policies: Which worker should process the next segment of a model if early-exit is not triggered. This is managed using input and output queues, deciding whether local processing or offloading to a neighbor is faster.
  • Data Admission Policies: The source worker adaptively adjusts the data arrival rate (or equivalently, the interarrival time of data) based on the current queue occupancy of the system. This ensures that the system doesn't get overwhelmed.
  • Confidence Level Adaptation: The early-exit threshold itself can be adapted based on queue occupancy, allowing for higher accuracy when the system is less loaded and lower accuracy (more early exits) when it's overloaded to maintain throughput.

The framework is designed to be decentralized, with each worker making local decisions based on its queue status and neighbor information.

Datasets

  • CIFAR-10: Used for evaluating the performance of the proposed framework.
  • ImageNet: Mentioned as a source for the DNN models used.

The paper does not introduce new benchmarks but rather evaluates existing DNN models on standard datasets within a custom edge testbed.

Key Insights and Ideas

  • Early-Exit is a Natural Fit for MDI: The paper elegantly combines two concepts that address edge computing limitations: MDI distributes the computational load, and early-exit reduces unnecessary computation for simpler inputs.
  • Adaptive Control is Crucial: The success of MDI-Exit hinges on its adaptive policies for offloading, data admission, and early-exit thresholds. This allows the system to dynamically adjust to varying network conditions and data complexities.
  • Trade-offs are Explicitly Managed: The framework allows for explicit trade-offs between data arrival rate and inference accuracy. By adjusting the early-exit threshold, the system can prioritize throughput or precision.
  • Topology Matters: The performance of MDI-Exit can be influenced by the network topology. While more nodes generally offer more resources, communication bottlenecks can arise in highly connected topologies (like a 5-node mesh) if not managed carefully.

Key Results

  • Increased Data Processing: MDI-Exit processes more data when accuracy is fixed compared to baseline methods.
  • Higher Accuracy: MDI-Exit achieves higher accuracy for a fixed data rate.
  • Topology Benefits: In multi-node setups, MDI-Exit achieves a higher data arrival rate by utilizing parallel computing resources.
  • Auto-encoder Impact: For ResNet-50, an auto-encoder was used after the first exit point to reduce feature vector size, leading to a slight accuracy drop (up to 2.2%) but significantly reducing transmission delays.
  • Experimental Validation: The framework was tested on a real-world testbed of NVIDIA Jetson Nano edge devices, demonstrating its practical applicability.

Limitations and Future Work

  • Auto-encoder Accuracy Trade-off: The paper notes that using an auto-encoder for feature compression can slightly reduce accuracy.
  • Topology Sensitivity: Performance can vary with topology, with communication bottlenecks becoming an issue in highly meshed networks.
  • Future Work: The authors suggest that MDI-Exit's performance can be further enhanced by incorporating other DNN optimization techniques like conditional architectures, pruning, and quantization. The paper also implicitly points towards exploring more sophisticated topology management and communication protocols for even greater efficiency.

Summary updated

Abstract

Distributed inference techniques can be broadly classified into data-distributed and model-distributed schemes. In data-distributed inference (DDI), each worker carries the entire deep neural network (DNN) model but processes only a subset of the data. However, feeding the data to workers results in high communication costs, especially when the data is large. An emerging paradigm is model-distributed inference (MDI), where each worker carries only a subset of DNN layers. In MDI, a source device that has data processes a few layers of DNN and sends the output to a neighboring device, i.e., offloads the rest of the layers. This process ends when all layers are processed in a distributed manner. In this paper, we investigate the design and development of MDI with early-exit, which advocates that there is no need to process all the layers of a model for some data to reach the desired accuracy, i.e., we can exit the model without processing all the layers if target accuracy is reached. We design a framework MDI-Exit that adaptively determines early-exit and offloading policies as well as data admission at the source. Experimental results on a real-life testbed of NVIDIA Nano edge devices show that MDI-Exit processes more data when accuracy is fixed and results in higher accuracy for the fixed data rate.
BibTeX
@inproceedings{colocrese2024early,
  author = {Marco Colocrese and Erdem Koyuncu and Hulya Seferoglu},
  title = {{Early-Exit Meets Model-Distributed Inference at Edge Networks}},
  booktitle = {LANMAN},
  year = {2024},
  url = {https://openreview.net/forum?id=dUjZRbQgd4},
}
Figure