URNet : User-Resizable Residual Networks with Conditional Gating Module

Sang-ho Lee, Simyung Chang, Nojun Kwak

TL;DR

URNet introduces a "volume knob" for deep neural networks, allowing users to dynamically adjust a model's computational cost at inference time. By integrating lightweight Conditional Gating Modules (CGMs) into ResNets, the architecture learns to skip redundant residual blocks based on both the input image and a user-defined scale parameter, maintaining high accuracy even when significantly reduced in size.

What Problem Does This Paper Address?

Standard Convolutional Neural Networks (CNNs) are computationally expensive and "static"—once trained, their computational cost is fixed regardless of the hardware environment or the urgency of a request. While techniques like pruning or dynamic path selection exist, they have major drawbacks:

  1. Static Compression: Methods like pruning create a single, smaller model that cannot scale back up if resources become available.
  2. Lack of User Control: Existing dynamic networks change their path based on the input (e.g., skipping layers for "easy" images), but they don't allow a user or system administrator to throttle the network's total resource usage during a traffic spike.
  3. Training Instability: Many dynamic architectures rely on Reinforcement Learning (RL), which is notoriously slow and unstable to train.

URNet solves these by providing a user-resizable framework that is trained using simple, stable supervised learning.

Methods

The core of URNet is the Conditional Gating Module (CGM). A CGM is placed before each residual block to decide whether that block should be executed or skipped.

Figure 2: Overall structure of URNet. URNet locates Conditional Gating Module (CGM) for each residual block of ResNet, and de- termines whether each block is used or not. CGMs receive two inputs and output one gate value: one of the inputs is the features of the previous layer and the other is the desired scale parameter S. To output a gate value, it uses a sigmoid function with probability p and a binary step function with probability 1 −p as an activation function. CGMs, with sigmoid function, can be trained through scale loss Ls and classification loss Lc to increase classification performance while controlling the number of blocks used. At the time of inference, p is set to 0, so that only the binary step is used and the amount of computation is reduced by not using the blocks with the gate value of 0.

The Gating Mechanism

Each CGM takes two inputs: the features from the previous layer and a user-specified scale parameter $S$ (ranging from 0 to 1).

  • Architecture: The module uses global average pooling on the features, concatenates them with the scale $S$, and passes them through a few fully connected layers to produce a single gate value.
  • Training Trick: To handle the non-differentiable nature of a binary "on/off" switch, the authors use a probabilistic approach. During training, the gate acts as a differentiable sigmoid function with probability $p$ and a binary step function with probability $1-p$. At inference, $p$ is set to 0, making the gates purely binary.

The Scale Loss

To ensure the network actually follows the user's requested scale, the authors introduce a scale loss ($L_s$): $$L_s = \left( \frac{1}{N} \sum_{n=1}^{N} gate_n - S \right)^2$$ This loss penalizes the network if the average number of active blocks deviates from the target scale $S$. The total loss is a weighted sum of this scale loss and standard classification loss.

Datasets

The authors evaluated URNet across several standard benchmarks:

  • CIFAR-10 & CIFAR-100: Used with a ResNet-110 baseline.
  • ImageNet (ILSVRC2012): Used with a ResNet-101 baseline.
  • Pascal VOC 2007: Used to demonstrate the method's extensibility to object detection tasks (Faster R-CNN).

Key Insights and Ideas

  • User-Controllable Sparsity: Unlike previous works, the "knob" is external. A server can dynamically lower $S$ for all incoming requests to maintain latency during peak hours without reloading a different model.
  • Input-Adaptive Importance: Even at a fixed scale $S$, the network doesn't just pick blocks at random. It learns which blocks are critical for specific images. "Hard" samples (noisy or cluttered images) naturally trigger more blocks than "easy" ones.
  • ResNet as an Ensemble: The work builds on the intuition that ResNets behave like ensembles of shallower networks. Dropping layers doesn't break the network; it just slightly reduces the "ensemble" size.
  • Supervised Efficiency: By avoiding Reinforcement Learning, URNet trains significantly faster. For CIFAR, URNet requires only 500 epochs compared to 7,000 epochs for RL-based competitors like BlockDrop.
Figure 4: The block usage map of URNet-110 on the CIFAR-10 test set with β = 4.0. The horizontal axis is the scale parameter S and the vertical axis is the index of 54 residual blocks. As S increases, the usage of blocks gradually increases. Also, the pres- ence of blocks whose usage is not 1 or 0, means that the usage of the block varies according to the input image even on the same scale.

Key Results

  • ImageNet Performance: URNet based on ResNet-101 maintains baseline accuracy (76.4%) even when resized to ~80% of the original blocks. It shows only a ~1% accuracy drop when using just 65% of the total computation.
  • Regularization Effect: On CIFAR datasets, URNet sometimes outperforms the baseline ResNet at scales of 0.7 or 0.8. The authors attribute this to a regularization effect similar to stochastic depth or dropout.
  • Object Detection: When applied to Faster R-CNN, URNet successfully reduced computation while maintaining competitive mAP (mean Average Precision), proving the module is versatile.

Limitations and Future Work

  • Range vs. Performance Trade-off: There is a natural trade-off; training a model to be highly accurate across a very wide range of $S$ (e.g., 0.1 to 1.0) is harder than training for a narrow range.
  • Architecture Specificity: The current implementation is tailored for Residual Networks. While ResNets are ubiquitous, extending this to other architectures like Vision Transformers (ViTs) or DenseNets would be a logical next step.
  • Extreme Sparsity: At very low scales ($S < 0.2$), accuracy begins to degrade more sharply as the network loses the minimum depth required for complex feature extraction.

Summary updated

Abstract

Convolutional Neural Networks are widely used to process spatial scenes, but their computational cost is fixed and depends on the structure of the network used. There are methods to reduce the cost by compressing networks or varying its computational path dynamically according to the input image. However, since a user can not control the size of the learned model, it is difficult to respond dynamically if the amount of service requests suddenly increases. We propose User-Resizable Residual Networks (URNet), which allows users to adjust the scale of the network as needed during evaluation. URNet includes Conditional Gating Module (CGM) that determines the use of each residual block according to the input image and the desired scale. CGM is trained in a supervised manner using the newly proposed scale loss and its corresponding training methods. URNet can control the amount of computation according to user's demand without degrading the accuracy significantly. It can also be used as a general compression method by fixing the scale size during training. In the experiments on ImageNet, URNet based on ResNet-101 maintains the accuracy of the baseline even when resizing it to approximately 80% of the original network, and demonstrates only about 1% accuracy degradation when using about 65% of the computation.
BibTeX
@inproceedings{lee2020urnet,
  author = {Sang-ho Lee and Simyung Chang and Nojun Kwak},
  title = {{URNet : User-Resizable Residual Networks with Conditional Gating Module}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5886},
}
Figure