Stimulative Training++: Go Beyond the Performance Limits of Residual Networks

Peng Ye, Tong He, Shengji Tang, Baopu Li, Tao Chen, Lei Bai, Wanli Ouyang

Introduction

Residual Networks (ResNets) are widely understood as an ensemble of shallow subnetworks rather than a single deep pipeline. While this "unraveled view" explains their robustness, it also masks a significant inefficiency. Researchers have identified a phenomenon termed "network loafing"—a concept borrowed from social psychology where individuals exert less effort when working in a group. In ResNets, internal subnetworks perform significantly worse than they would if trained independently, creating a bottleneck for the overall model's potential.

The "Social Loafing" Problem

The core observation is that subnetworks within a large ResNet are "lazy." For instance, a ResNet20 subnetwork embedded within a trained ResNet32 achieves only ~55% accuracy on CIFAR-10, whereas a standalone ResNet20 typically reaches ~92%. This performance gap is not unique to ResNets; it persists across various architectures, including DenseNets and Vision Transformers (ViTs), suggesting that collective training naturally leads to individual subnetwork underperformance.

Figure 1: Evidence of "social loafing" across different architectures. Subnetworks (points on the curves) consistently underperform compared to their standalone counterparts (stars at the top left of each series).

Methodology: Stimulative Training++ (ST++)

To combat this, the authors propose Stimulative Training++ (ST++), a framework designed to provide "individual accountability" for subnetworks during the training process.

1. Targeted Subnetwork Supervision

ST++ randomly samples a subnetwork in each iteration and applies a specialized KL- loss. Unlike standard Knowledge Distillation, KL- aligns only the direction of the predicted class probabilities (the "intent"), rather than the magnitude of the logits. This allows subnetworks of different sizes to maintain their own confidence levels while still learning from the main network's predictions.

2. Multi-Scale Input and Receptive Fields

Shallow subnetworks naturally have smaller Effective Receptive Fields (ERF). To compensate, ST++ utilizes random downscaling resolution. By feeding subnetworks smaller images, their relative receptive field increases, allowing them to capture global context similar to what the deeper main network achieves through more layers.

3. Ordered Residual Sampling

To manage the exponential number of possible subnetworks ($2^n$), ST++ employs an ordered sampling rule. This maintains the residual structure and reduces the search space to $O(n)$, making the training process computationally feasible without sacrificing the diversity of stimulated parts.

Figure 2: The ST++ training scheme, featuring random downscaling, inter- and inner-stage sampling, and the direction-only KL- loss.

Key Results and Benchmarks

ST++ achieves state-of-the-art results without modifying model architectures or using external data:

  • ImageNet Performance: ST++ pushed a standard ResNet50 to 81.0% Top-1 accuracy, a record-breaking milestone for this architecture. It outperforms highly optimized training recipes from popular libraries like timm and official PyTorch releases.
  • Downstream Tasks: Models pretrained with ST++ show superior generalization. On MS COCO, using an ST++ backbone for Faster R-CNN or Mask R-CNN yielded up to a 2.1% boost in Bbox mAP over standard baselines.
  • Efficiency: ST++ is remarkably efficient; a model trained for only 50 epochs with ST++ can outperform a standard model trained for 200 epochs.

Conclusion

Stimulative Training++ demonstrates that the performance of deep networks is often limited by the underperformance of their internal components. By addressing "network loafing" through targeted supervision and resolution scaling, ST++ unlocks the latent potential of established architectures like ResNet, DenseNet, and ViT, setting a new standard for efficient and effective training in computer vision.

Summary updated

Abstract

Residual networks have shown great success and become indispensable in recent deep neural network models. In this work, we aim to re-investigate the training process of residual networks from a novel social psychology perspective of loafing, and further propose a new training scheme as well as three improved strategies for boosting residual networks beyond their performance limits. Previous research has suggested that residual networks can be considered as ensembles of shallow networks, which implies that the final performance of a residual network is influenced by a group of subnetworks. We identify a previously overlooked problem that is analogous to social loafing, where subnetworks within a residual network are prone to exert less effort when working as part of a group compared to working alone. We define this problem as \textit{network loafing}. Similar to the decreased individual productivity and overall performance as demonstrated in society, network loafing inevitably causes sub-par performance. Inspired by solutions from social psychology, we first propose a novel training scheme called stimulative training, which randomly samples a residual subnetwork and calculates the KL divergence loss between the sampled subnetwork and the given residual network for extra supervision. In order to unleash the potential of stimulative training, we further propose three simple-yet-effective strategies, including a novel KL- loss that only aligns the network logits direction, random smaller inputs for subnetworks, and inter-stage sampling rules. Comprehensive experiments and analysis verify the effectiveness of stimulative training as well as its three improved strategies.
BibTeX
@article{ye2023stimulative,
  author = {Peng Ye and Tong He and Shengji Tang and Baopu Li and Tao Chen and Lei Bai and Wanli Ouyang},
  title = {{Stimulative Training++: Go Beyond the Performance Limits of Residual Networks}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
}
Figure