Abstract
With the capability of trading accuracy for latency on-the-fly, the technique of adaptive early-exit inference has emerged as a promising line of research to accelerate the deep learning inference. However, studies in this line of research commonly use a group of thresholds to control the accuracy-latency trade-off, where a thorough and general methodology on how to determine these thresholds has not been conducted yet, especially with regard to the common requirements of average inference latency. To address this issue and enable latency-aware adaptive early-exit inference, in the present paper, we approximately formulate the threshold determination problem of finding the accuracy-maximum threshold setting that meets a given average latency requirement, and then propose a threshold determination method to tackle our formulated non-convex problem. Theoretically, we prove that, for certain parameter settings, our method finds an approximate stationary point of the formulated problem. Empirically, on top of various models across multiple datasets (CIFAR-10, CIFAR-100, ImageNet and two time-series datasets), we show that our method can well handle the average latency requirements, and consistently finds good threshold settings in negligible time.
Introduction
Deep neural networks (DNNs) are increasingly deployed on resource-constrained devices like mobile phones and IoT sensors. However, the high computational cost of modern DNNs often clashes with strict latency requirements. Adaptive early-exit inference offers a solution by introducing "exit branches" at intermediate layers. This allows "easy" samples to be classified early, saving computation, while "hard" samples proceed through the full network.
The primary challenge in this approach is determining the optimal confidence thresholds for each exit. Currently, this is often a "black box" process relying on manual heuristics or simple grid searches. These methods fail to provide a systematic way to meet specific average latency requirements and often struggle with complex, multi-exit architectures like LSTMs or DenseNets where exit behaviors vary significantly.
Methodology
This research transforms threshold determination from a manual tuning task into a formal mathematical optimization problem. The authors propose a data-driven framework consisting of several key components:
- Differentiable Approximation: The binary decision to exit is inherently non-differentiable. To enable gradient-based optimization, the authors employ the Verhulst logistic function to create a smooth approximation of the exit process.
- Constrained Optimization: The problem is formulated to maximize classification accuracy while treating the average latency as a hard constraint.
- The iPPP Algorithm: To solve the resulting non-convex problem, the authors use the inexact proximal-point penalty (iPPP) method. This approach uses an outer loop to manage the latency penalty and an inner loop utilizing Nesterov’s Accelerated Proximal Gradient (APG) to optimize thresholds.
- Warm-Start Strategy: A line-search strategy is used to find a feasible initial threshold setting, ensuring the optimization begins from a point that already satisfies the latency constraints.
Experimental Evaluation
The method was evaluated across a diverse set of architectures and data types:
- Computer Vision: CIFAR-10 (B-AlexNet), CIFAR-100 (S-ResNet-18), and ImageNet (MSDNet).
- Time-Series/Speech: Google-30 (speech commands) and DSA-19 (daily and sports activities) using LSTM models.
The inclusion of LSTMs is particularly significant, as recurrent networks can have dozens of potential exit points, making manual threshold tuning practically impossible.
Results and Discussion
The proposed method demonstrates significant improvements in both accuracy and latency adherence compared to standard heuristics.
As shown in the figure above, the proposed method (solid green and dashed pink lines) strictly adheres to the latency budget (following the diagonal in subplot b) while maintaining significantly higher accuracy than baselines, particularly in scenarios where traditional heuristics fail to scale or explore the full trade-off space.
Key Insights
- Trainable Thresholds: By approximating exit logic with smooth functions, thresholds can be "trained" on a validation set similarly to model weights, leading to more precise control.
- Scalability to Many Exits: The automated approach excels in models with many exit points. For instance, it achieved accuracy gains of up to 9.01% on the Google-30 dataset.
- Strict Latency Awareness: Unlike previous methods, this algorithm allows engineers to specify a precise latency budget, and it automatically finds the highest possible accuracy within that constraint.
Quantitative Gains
On S-ResNet-18 (CIFAR-100), the method improved accuracy by 0.79% over standard heuristics. On LSTM models (DSA-19), the gains were more dramatic, outperforming the "single-threshold" baseline by over 7%. Furthermore, the optimization process is highly efficient, taking only 0.5s to 8.6s to find optimal thresholds.
Conclusion and Future Work
This work provides a robust, automated framework for latency-aware early-exit inference. While highly effective, the authors identify a "generalization gap" if test data is significantly more complex than validation data. Future research will focus on reducing the number of hyperparameters in the iPPP algorithm and addressing "tail latency" for real-time systems where worst-case performance is critical.
Summary updated