Dynamic Representations Toward Efficient Inference on Deep Neural Networks by Decision Gates

Mohammad Saeed Shafiee, Alexander WongWorkshops @ CVPR 2019 · OpenReview

Abstract: While deep neural networks extract rich features from the input data, the current trade-off between depth and computational cost makes it difficult to adopt deep neural networks for many industrial applications, especially when computing power is limited. Here, we are inspired by the idea that, while deeper embeddings are needed to discriminate difficult samples (i.e., fine-grained discrimination), a large number of samples can be well discriminated via much shallower embeddings (i.e., coarse-grained discrimination). In this study, we introduce the simple yet effective concept of decision gates (d-gate), modules trained to decide whether a sample needs to be projected into a deeper embedding or if an early prediction can be made at the d-gate, thus enabling the computation of dynamic representations at different depths. The proposed d-gate modules can be integrated with any deep neural network and reduces the average computational cost of the deep neural networks while maintaining modeling accuracy. The proposed d-gate framework is examined via different network architectures and datasets, with experimental results showing that leveraging the proposed d-gate modules led to a 43% speed-up and 44% FLOPs reduction on ResNet-101 and 55% speed-up and 39% FLOPs reduction on DenseNet-201 trained on the CIFAR10 dataset with only 2% drop in accuracy. Furthermore, experiments where d-gate modules are integrated into ResNet-101 trained on the ImageNet dataset demonstrate that it is possible to reduce the computational cost of the network by 1.5 GFLOPs without any drop in the modeling accuracy.

Introduction

Deep Neural Networks (DNNs) have achieved state-of-the-art performance by increasing architectural depth, but this comes at a significant computational cost. In standard DNN architectures, every input—regardless of its complexity—undergoes the same fixed amount of computation. This uniform processing is inherently inefficient. Drawing inspiration from human perception, which can recognize coarse categories almost instantly while requiring more focus for fine-grained details, this paper proposes a mechanism for dynamic inference.

The core objective is to build a system that can reliably decide when a sample has been sufficiently processed to allow for an "early exit," thereby bypassing unnecessary deeper layers without sacrificing overall model accuracy.

The Decision Gate Framework

The authors introduce the Decision Gate (d-gate), a lightweight, single-layer feed-forward module. These gates are strategically integrated after major computational blocks, such as residual blocks in ResNet or dense blocks in DenseNet.

The d-gate framework integrated into a deep neural network, showing multiple early exit points.

During inference, a d-gate evaluates the intermediate representation of a sample. If the gate determines the sample is "easy" enough to classify, it triggers an early prediction and terminates further computation for that input. If the sample is deemed "hard," it is passed to the next block for deeper feature extraction. This creates a "mixture of depths" where easy images take an express lane and difficult ones take the full path through the network.

Leveraging Soft-Margin Hinge Loss

A critical innovation in this framework is the use of Soft-Margin Hinge Loss instead of the standard cross-entropy loss for training the d-gates. While cross-entropy is standard for classification, it does not inherently provide a reliable measure of confidence based on the decision boundary.

Illustration of easier vs. harder samples based on their distance from the decision boundary.

By using Hinge Loss—the mathematical foundation of Support Vector Machines (SVMs)—the d-gate can calculate the distance of a sample to the decision boundaries. This distance serves as a proxy for confidence:

  1. Thresholding: A predefined threshold $t$ is set.
  2. Decision: If the maximum distance to a boundary exceeds $t$, the model makes an early prediction.
  3. Efficiency: Because d-gates are simple single layers, their computational overhead is negligible compared to the massive convolutional blocks they allow the model to skip.

Experimental Results

The framework was evaluated on CIFAR-10 and ImageNet (ILSVRC2012) using ResNet-101 and DenseNet-201 architectures. The results demonstrate significant efficiency gains:

  • CIFAR-10: ResNet-101 achieved a 43% speedup and 44% reduction in FLOPs with only a 2% accuracy drop. DenseNet-201 saw even higher gains with a 55% speedup.
  • ImageNet: On this large-scale dataset, the d-gate integration reduced computational cost by 1.5 GFLOPs with zero drop in accuracy.
  • Qualitative Analysis: Shallow d-gates were found to successfully classify images with distinguishable features, while "harder" classes (e.g., specific animal breeds) consistently required deeper layers for fine-grained feature extraction.

Conclusion and Future Work

The d-gate framework provides a simple yet effective method for dynamic inference. While currently trained independently of the main network, future work may explore joint optimization to further refine performance. Additionally, the authors suggest that a hierarchical approach—where early gates identify broad categories and deeper layers refine them—could further enhance the efficiency of deep learning models in resource-constrained environments.

Summary updated

Abstract

While deep neural networks extract rich features from the input data, the current trade-off between depth and computational cost makes it difficult to adopt deep neural networks for many industrial applications, especially when computing power is limited. Here, we are inspired by the idea that, while deeper embeddings are needed to discriminate difficult samples (i.e., fine-grained discrimination), a large number of samples can be well discriminated via much shallower embeddings (i.e., coarse-grained discrimination). In this study, we introduce the simple yet effective concept of decision gates (d-gate), modules trained to decide whether a sample needs to be projected into a deeper embedding or if an early prediction can be made at the d-gate, thus enabling the computation of dynamic representations at different depths. The proposed d-gate modules can be integrated with any deep neural network and reduces the average computational cost of the deep neural networks while maintaining modeling accuracy. The proposed d-gate framework is examined via different network architectures and datasets, with experimental results showing that leveraging the proposed d-gate modules led to a 43% speed-up and 44% FLOPs reduction on ResNet-101 and 55% speed-up and 39% FLOPs reduction on DenseNet-201 trained on the CIFAR10 dataset with only 2% drop in accuracy. Furthermore, experiments where d-gate modules are integrated into ResNet-101 trained on the ImageNet dataset demonstrate that it is possible to reduce the computational cost of the network by 1.5 GFLOPs without any drop in the modeling accuracy.
BibTeX
@inproceedings{shafiee2019dynamic,
  author = {Mohammad Saeed Shafiee and Mohammad Javad Shafiee and Alexander Wong},
  title = {{Dynamic Representations Toward Efficient Inference on Deep Neural Networks by Decision Gates}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=fHiiNT9OHI},
}
Figure