RDI-Net: Relational Dynamic Inference Networks

Huanyu Wang, Songyuan Li, Shihao Su, Zequn Qin, Xi LiICCV 2021 · OpenReview

Abstract

Dynamic inference networks, aimed at promoting computational efficiency, go along an adaptive executing path for a given sample. Prevalent methods typically assign a router for each convolutional block and sequentially make block-by-block executing decisions, without considering the relations during the dynamic inference. In this paper, we model the relations for dynamic inference from two aspects: the routers and the samples. We design a novel type of router called the relational router to model the relations among routers for a given sample. In principle, the current relational router aggregates the contextual features of preceding routers by graph convolution and propagates its router features to subsequent ones, making the executing decision for the current block in a long-range manner. Furthermore, we model the relation between samples by introducing a Sample Relation Module (SRM), encouraging correlated samples to go along correlated executing paths. As a whole, we call our method the Relational Dynamic Inference Network (RDI-Net). Extensive experiments on CIFAR-10/100 and ImageNet show that RDI-Net achieves state-of-the-art performance and computational cost reduction.

Introduction

Dynamic inference is a popular strategy for making deep learning more efficient: instead of running every layer for every image, the network "skips" certain blocks for "easier" samples. However, most existing methods are "short-sighted." They typically assign a "router" to each convolutional block that decides whether to execute that block based only on the output of the immediately preceding layer.

The authors identify two major gaps in this approach:

  1. Router Isolation: Decisions are made in a short-range, Markovian fashion, ignoring the context of decisions made earlier in the pipeline.
  2. Sample Isolation: The network treats every image as an independent event, failing to exploit the intuition that visually similar samples should likely follow similar processing paths.
Comparison between prevalent dynamic inference methods and the proposed RDI-Net architecture.

Methods

RDI-Net introduces two primary modules to bridge these gaps:

  • Relational Routers: Instead of a simple decision gate, each router employs a router-wise directed relation graph. Using Graph Convolutional Networks (GCNs), a router aggregates features from $r$ preceding routers. This allows the network to make "long-range" decisions. The influence of previous routers decays as the distance increases, modeled by a decay rate $\gamma$.
  • Sample Relation Module (SRM): This module enforces "path consistency." It calculates a distance matrix for samples within a training batch. Using a specialized triplet loss, it regularizes the network so that the execution paths of similar samples are closer to each other than those of dissimilar samples.
  • Uniform Sampling Warm-up (USW): To prevent the network from falling into local optima (where routers might prematurely learn to skip blocks that haven't been fully trained), the authors use a two-stage training process. They first "warm up" the backbone by randomly sampling execution paths before training the relational routers.
Detailed architecture of the Relational Router, illustrating feature propagation among routers and the final executing decision process.

Key Results and Insights

The authors evaluated RDI-Net on standard image classification benchmarks including CIFAR-10/100 and ImageNet (ILSVRC2012). RDI-Net demonstrates impressive efficiency gains across all tested datasets:

  • CIFAR-10: Reduced computational cost by 73% while actually increasing accuracy by 0.13% compared to a standard ResNet-110.
  • CIFAR-100: Reduced cost by 65% with a 0.42% accuracy gain.
  • ImageNet: On ResNet-50, it achieved a 35% reduction in GFLOPs while maintaining competitive accuracy, outperforming other dynamic methods like Conv-AIG and SACT.

Decisions Benefit from Context: The paper finds that a router's decision is more robust when it knows what the previous routers "saw." Looking back at roughly 6 preceding routers provides the optimal balance of context versus semantic noise. Furthermore, by forcing similar images into similar paths, the SRM acts as a powerful regularizer, ensuring the dynamic behavior of the network is grounded in the underlying data distribution.

Conclusion

RDI-Net optimizes neural network efficiency by moving beyond isolated, block-by-block "skip or execute" decisions. By modeling long-range dependencies between decision-making routers and ensuring that similar images follow similar execution paths, it achieves state-of-the-art performance with significantly reduced computational costs. Future work could explore more sophisticated self-supervised embeddings to further improve sample-to-path correlation and explore even more efficient ways to propagate router features.

Summary updated

Abstract

Dynamic inference networks, aimed at promoting computational efficiency, go along an adaptive executing path for a given sample. Prevalent methods typically assign a router for each convolutional block and sequentially make block-by-block executing decisions, without considering the relations during the dynamic inference. In this paper, we model the relations for dynamic inference from two aspects: the routers and the samples. We design a novel type of router called the relational router to model the relations among routers for a given sample. In principle, the current relational router aggregates the contextual features of preceding routers by graph convolution and propagates its router features to subsequent ones, making the executing decision for the current block in a long-range manner. Furthermore, we model the relation between samples by introducing a Sample Relation Module (SRM), encouraging correlated samples to go along correlated executing paths. As a whole, we call our method the Relational Dynamic Inference Network (RDI-Net). Extensive experiments on CIFAR-10/100 and ImageNet show that RDI-Net achieves state-of-the-art performance and computational cost reduction.
BibTeX
@inproceedings{wang2021rdi,
  author = {Huanyu Wang and Songyuan Li and Shihao Su and Zequn Qin and Xi Li},
  title = {{RDI-Net: Relational Dynamic Inference Networks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=6RXuuq_Pn9Z},
}
Figure