Dynamic Routing Networks

Shaofeng Cai, Yao Shu, Wei WangWACV 2021 · OpenReview

Abstract: The deployment of deep neural networks in real-world applications is mostly restricted by their high inference costs. Extensive efforts have been made to improve the accuracy with expert-designed or algorithm-searched architectures. However, the incremental improvement is typically achieved with increasingly more expensive models that only a small portion of input instances really need. Inference with a static architecture that processes all input instances via the same transformation would thus incur unnecessary computational costs. Therefore, customizing the model capacity in an instance-aware manner is much needed for higher inference efficiency. In this paper, we propose Dynamic Routing Networks (DRNets), which support efficient instance-aware inference by routing the input instance to only necessary transformation branches selected from a candidate set of branches for each connection between transformation nodes. The branch selection is dynamically determined via the corresponding branch importance weights, which are first generated from lightweight hyper-networks (RouterNets) and then recalibrated with Gumbel-Softmax before the selection. Extensive experiments show that DRNets can reduce a substantial amount of parameter size and FLOPs during inference with prediction performance comparable to state-of-the-art architectures.

Introduction

Traditional deep learning models employ a static architecture where every input instance—regardless of its complexity—is processed through the same sequence of operations. This "one-size-fits-all" approach is computationally inefficient, as simpler inputs often require significantly less model capacity than difficult ones. While Neural Architecture Search (NAS) has produced efficient static models, they lack the ability to adapt dynamically during inference. Dynamic Routing Networks (DRNets) address this by enabling instance-aware inference, where the network path is tailored to the specific input.

Methods: Architecture and Routing

DRNets consist of a backbone network composed of sequential "Cells." Within each cell, nodes are connected by multiple candidate branches, such as various convolution sizes, pooling operations, or identity skip connections.

Figure 1: (a) Internal cell structure with nodes and (b) the RouterNet-driven node connection mechanism.

The core innovation is the RouterNet, a lightweight hyper-network integrated into each cell. RouterNets analyze the input features to predict importance weights for each available branch. To make the discrete choice of a branch differentiable for training, the authors utilize the Gumbel-Softmax trick. This allows the model to be trained using standard backpropagation by providing a continuous relaxation of the routing decision.

Two-Stage Training Strategy

To optimize both accuracy and efficiency, DRNets are trained in two stages:

  1. Exploration: The model is trained with a high Gumbel-Softmax temperature, encouraging it to explore and learn from all available branches.
  2. Specialization: The temperature is gradually annealed, and a resource regularizer is added to the loss function. This regularizer penalizes high FLOP usage, forcing the network to favor lighter branches (like skip connections) for easier inputs.

Key Insights and Results

DRNets demonstrate that instance-awareness is a powerful tool for efficiency. Visualizations show that the model naturally learns to use fewer, lighter branches for high-contrast, clear images while reserving complex convolutions for more challenging, low-quality inputs.

Figure 5: Selection ratios of different operations across network cells, showing the dynamic distribution of computational resources.

Experimental results on CIFAR-10 and ImageNet highlight the efficiency gains:

  • Efficiency: On CIFAR-10, DRNets achieved a 3.71x reduction in parameters and a 4.53x reduction in FLOPs compared to the DARTS architecture while maintaining comparable accuracy.
  • Scalability: On ImageNet, the DRNet(L) variant outperformed established models like MobileNetV2 in accuracy-per-FLOP metrics, utilizing only 351M FLOPs.
  • Training Speed: Unlike Reinforcement Learning-based dynamic models, DRNets can be trained in just 2.5 to 5.5 GPU days, making them significantly more practical for research and deployment.

Conclusion and Future Work

DRNets provide a robust framework for instance-aware deep learning, significantly lowering the barrier for deploying high-performance models on resource-constrained hardware. Future research may focus on integrating more complex network topologies and optimizing the resource regularizer for hardware-specific metrics like actual latency or Memory Access Cost (MAC) on edge devices.

Summary updated

Abstract

The deployment of deep neural networks in real-world applications is mostly restricted by their high inference costs. Extensive efforts have been made to improve the accuracy with expert-designed or algorithm-searched architectures. However, the incremental improvement is typically achieved with increasingly more expensive models that only a small portion of input instances really need. Inference with a static architecture that processes all input instances via the same transformation would thus incur unnecessary computational costs. Therefore, customizing the model capacity in an instance-aware manner is much needed for higher inference efficiency. In this paper, we propose Dynamic Routing Networks (DRNets), which support efficient instance-aware inference by routing the input instance to only necessary transformation branches selected from a candidate set of branches for each connection between transformation nodes. The branch selection is dynamically determined via the corresponding branch importance weights, which are first generated from lightweight hyper-networks (RouterNets) and then recalibrated with Gumbel-Softmax before the selection. Extensive experiments show that DRNets can reduce a substantial amount of parameter size and FLOPs during inference with prediction performance comparable to state-of-the-art architectures.
BibTeX
@inproceedings{cai2021dynamic,
  author = {Shaofeng Cai and Yao Shu and Wei Wang},
  title = {{Dynamic Routing Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=dGyV1qieXXz},
}
Figure