Abstract
Deep Neural Networks (DNNs) are computationally intensive, often processing simple and complex inputs with the same level of depth. This paper introduces Soft-Guided Adaptively-Dropped (SGAD), an input-adaptive framework for Residual Networks (ResNets) that dynamically skips redundant convolutional blocks based on sample difficulty. By utilizing a temporary "soft guideline" network during training to supervise a lightweight mask-generating network, SGAD reduces computational costs (FLOPs) by up to 77% with less than a 1% drop in accuracy. Notably, the guideline network is discarded after training, ensuring zero inference overhead.
Introduction
While Deep Neural Networks have achieved state-of-the-art performance in computer vision, their high computational requirements hinder deployment on resource-constrained devices like mobile phones or IoT sensors. Traditional compression techniques—such as pruning or weight decomposition—are typically "static," applying the same reduction to every input.
However, input samples vary in complexity. A clear, centered image may require significantly less processing than a blurry or occluded one. SGAD addresses this inefficiency by adaptively allocating computation, ensuring that "easy" samples bypass unnecessary layers while "hard" samples utilize the full depth of the network.
Methodology
The SGAD architecture transforms a static ResNet into an adaptive one by introducing a mechanism to skip specific residual blocks. This is based on the observation that ResNets behave like an ensemble of shallow paths, making them resilient to layer skipping.
Core Components
- BMNet (Binary Mask Network): A tiny sub-network that generates a binary mask (0 or 1) for each residual block. A mask of 0 triggers a skip via the residual bypass, while 1 executes the block.
- SGNet (Soft Guideline Network): A shallow "tutor" network used exclusively during training. It measures sample difficulty using the variance of its softmax output; high variance indicates high confidence (an easy sample).
- Mapping Strategy: An exponential function maps the SGNet’s confidence score to a target "drop ratio," which serves as a regularization term in the loss function, encouraging the BMNet to skip more blocks for easy images.
Training and Inference
During training, the SGNet provides "guidelines" that help the BMNet learn which blocks are most contributive for specific inputs. Once training is complete, the SGNet is discarded. During inference, only the BMNet remains active, allowing the model to make real-time skipping decisions with negligible overhead.
Key Insights
- Softmax Variance as a Metric: Using the variance of a shallow network's output provides a low-cost, automated way to quantify image difficulty without human labeling.
- Automated Pruning: The authors observed a "dead block" phenomenon where certain layers contribute very little to overall capacity. SGAD identifies and drops these layers for nearly all inputs, effectively performing automated architecture search.
- Adaptive Learning Rates: Mathematically, the binary mask acts as an adaptive learning rate for each block during training, focusing the network's optimization on the most critical layers.
Experimental Results
The researchers evaluated SGAD on the CIFAR-10 and CIFAR-100 benchmarks using ResNet-32 and ResNet-110 as baselines:
- CIFAR-10: On ResNet-110, SGAD achieved a 77% reduction in FLOPs with an accuracy drop of only 0.87%.
- CIFAR-100: On this more complex task, SGAD actually improved accuracy by 0.47% while reducing FLOPs by 23%.
- Comparative Performance: SGAD outperformed existing adaptive methods such as SACT, SkipNet, and BlockDrop in the accuracy-vs-computation trade-off.
Conclusion and Future Work
SGAD demonstrates that input-adaptive computation can significantly enhance the efficiency of ResNets. While FLOP reduction is a strong theoretical metric, the authors note that actual hardware speedups on GPUs can be limited by branching logic overhead. Future research will focus on scaling SGAD to high-resolution datasets like ImageNet and optimizing the framework for specific hardware architectures to translate FLOP savings into real-world latency improvements.
Summary updated