Introduction
Deep learning models are often over-parameterized for simple inputs, leading to unnecessary computational waste. Early-exit neural networks address this by adding side-branch classifiers that allow inference to terminate early if a high-confidence prediction is reached. However, traditional early-exit strategies suffer from a "granularity dilemma": frequent exit points (fine-grained) introduce significant overhead from redundant classifier checks, while sparse exit points (coarse-grained) miss opportunities for earlier termination.
Furthermore, existing methods are inherently "passive." The system only realizes it can exit once it reaches a branch and performs a check. This reactive nature prevents the system from optimizing hardware settings, such as Dynamic Voltage and Frequency Scaling (DVFS), during the inference process because the remaining workload is unknown.
The Predictive Exit Framework
To overcome these limitations, the "Predictive Exit" framework transforms early-exit networks into proactive decision-makers. Instead of checking every gate, the system forecasts the optimal exit point early in the inference process. The framework consists of three primary components:
- Unified Exiting Layer: To ensure predictable computation, all exit branches utilize a standardized, lightweight architecture. This consists of a Bag-of-Features (BoF) pooling layer followed by a Fully-Connected (FC) layer.
- Fine-Grained Exiting Points: By placing potential exits after every convolutional layer, the framework maximizes the potential for early termination without the usual overhead of passive checks.
- Low-Cost Prediction Engine: This core component uses a simple 1D convolution on intermediate features from an early layer to predict the final exit point. This prediction allows the model to skip all intermediate exit checks and execute only the predicted classifier.
Proactive Hardware Scaling
A key advantage of this predictive approach is the integration with DVFS. Once the engine predicts the exit layer (e.g., predicting an exit at layer 5), the system can calculate the exact remaining workload. It then proactively downclocks the processor to the lowest frequency that still meets the required latency deadline, significantly reducing power consumption.
Experimental Evaluation
The framework was evaluated using VGG-19 and ResNet-34 backbones across standard image classification benchmarks, including CIFAR-10, CIFAR-100, SVHN, and STL-10. Tests were conducted on both server-grade (NVIDIA GV100) and embedded (Jetson TX2) GPUs.
Key Results
- Computational Efficiency: The framework achieved up to a 96.2% reduction in computation compared to classic CNNs and outperformed state-of-the-art passive early-exit strategies by 12.8%.
- Energy Savings: Predictive Exit achieved up to 72.9% energy savings over standard models. On embedded hardware like the Jetson TX2, it demonstrated a 19.7% to 37.6% improvement in energy efficiency over existing early-exit methods.
- Accuracy Preservation: These significant efficiency gains were achieved while maintaining the same inference accuracy and meeting the same latency targets as high-performance baselines.
Discussion and Future Work
The "Predictive Exit" framework demonstrates that intermediate feature maps contain sufficient information to forecast classification confidence. By shifting from a passive to a proactive paradigm, the system eliminates the overhead of redundant checks and enables unprecedented hardware-level energy optimization.
However, some challenges remain. The framework currently requires a two-step training process for the backbone and exit layers. Additionally, while highly effective for CNNs, further research is needed to determine how these predictive dynamics apply to Transformer-based architectures. Future work will also explore automated hyperparameter tuning for the starting prediction layer and confidence thresholds across diverse hardware environments.
Summary updated