Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing

Peng Ye, Shengji Tang, Baopu Li, Tao Chen, Wanli Ouyang

Introduction

Residual Networks (ResNets) have become a cornerstone of modern deep learning, largely due to their ability to train extremely deep architectures via skip connections. A common framework for understanding these models is the "unraveled view," which posits that a ResNet is not merely a single deep chain but an ensemble of many shorter sub-networks or paths of varying lengths. While this ensemble nature provides redundancy and ease of optimization, it introduces a hidden inefficiency that this research terms "network loafing."

The Phenomenon of Network Loafing

Drawing inspiration from social psychology, the authors identify a parallel between human behavior in groups and the behavior of sub-networks within a ResNet. In psychology, "social loafing" describes the tendency for individuals to exert less effort when working collectively than when working alone.

The researchers observed a similar trend in deep learning: individual pathways within a large ResNet perform significantly worse than the same architectures trained in isolation. For instance, a ResNet-32 sub-network embedded within a ResNet-56 achieves only ~46% accuracy, whereas a standalone ResNet-32 typically achieves ~92%. This performance gap suggests that as the "group size" (network depth) increases, individual sub-networks rely too heavily on the collective ensemble, leading to fragile internal representations and poor structural robustness.

Stimulative Training (ST)

To combat network loafing, the authors propose Stimulative Training (ST), a strategy designed to force internal sub-networks to develop independent competence. The method is built upon two core principles derived from social psychology:

  1. Individual Accountability: During each training mini-batch, the algorithm randomly samples a specific sub-network from the larger ensemble. This ensures that individual paths are "seen" and evaluated by the loss function.
  2. Goal Consistency: The training objective incorporates a KL-divergence loss between the output of the sampled sub-network and the output of the full "main" network. By minimizing this divergence, the sub-network is forced to match the predictive power of the entire ensemble, effectively using the full network as a teacher for its own internal components.

Ordered Residual Sampling

A significant challenge in implementing ST is the exponential number of possible sub-networks ($2^n$). To make training computationally feasible, the authors introduce Ordered Residual Sampling. Instead of sampling arbitrary combinations of layers, the method skips a contiguous block of layers at the end of a specific stage. This reduces the sampling complexity to $O(n)$, maintains the mathematical properties of residual structures, and aligns with the theory of neural differential equations.

Experimental Evaluation

The authors evaluated Stimulative Training across several standard computer vision benchmarks, including CIFAR-10, CIFAR-100, and ImageNet, using architectures such as ResNet-50, ResNet-101, and MobileNetV3.

Accuracy and Performance

ST consistently outperformed standard training protocols:

  • CIFAR-100: ResNet-50 saw an accuracy increase of 4.43%, while MobileNetV3 improved by 3.68%.
  • ImageNet: On large-scale classification, ResNet-101 achieved a 1.21% improvement in Top-1 accuracy.
  • Sub-network Competence: The most dramatic gains were seen in the internal paths themselves. For MobileNetV3 on CIFAR-100, the mean accuracy of all internal sub-networks rose from 55.26% to 80.01%.

Structural Robustness

Because ST ensures that sub-networks are competent on their own, the resulting models exhibit remarkable "destruction resistance." In tests where layers were deleted or permuted at inference time, standard models experienced catastrophic performance collapses (e.g., a 35% drop for MobileNetV3). In contrast, models trained with ST maintained their accuracy with minimal degradation (less than 3% drop), demonstrating that the ensemble is no longer reliant on a single fixed path.

Discussion and Limitations

While Stimulative Training significantly improves accuracy and robustness, it introduces a trade-off in training efficiency. The requirement for an additional forward pass for the sampled sub-network increases training time by approximately 1.4x.

Furthermore, while the current study focuses on residual-based architectures, the authors hypothesize that network loafing likely persists in other ensemble-like structures, such as Transformers or DenseNets. Future work may involve adapting ST to these architectures or exploring its potential in self-supervised learning environments where explicit labels are unavailable.

Conclusion

By viewing neural network training through the lens of social psychology, Stimulative Training addresses the inherent "laziness" of sub-networks in deep architectures. By enforcing individual accountability and goal consistency, ST produces models that are not only more accurate but significantly more robust to structural perturbations, paving the way for more reliable deep learning systems.

Summary updated

Abstract

Residual networks have shown great success and become indispensable in today's deep models. In this work, we aim to re-investigate the training process of residual networks from a novel social psychology perspective of loafing, and further propose a new training strategy to strengthen the performance of residual networks. As residual networks can be viewed as ensembles of relatively shallow networks (i.e., \textit{unraveled view}) in prior works, we also start from such view and consider that the final performance of a residual network is co-determined by a group of sub-networks. Inspired by the social loafing problem of social psychology, we find that residual networks invariably suffer from similar problem, where sub-networks in a residual network are prone to exert less effort when working as part of the group compared to working alone. We define this previously overlooked problem as \textit{network loafing}. As social loafing will ultimately cause the low individual productivity and the reduced overall performance, network loafing will also hinder the performance of a given residual network and its sub-networks. Referring to the solutions of social psychology, we propose \textit{stimulative training}, which randomly samples a residual sub-network and calculates the KL-divergence loss between the sampled sub-network and the given residual network, to act as extra supervision for sub-networks and make the overall goal consistent. Comprehensive empirical results and theoretical analyses verify that stimulative training can well handle the loafing problem, and improve the performance of a residual network by improving the performance of its sub-networks. The code is available at https://github.com/Sunshine-Ye/NIPS22-ST .
BibTeX
@inproceedings{ye2022stimulative,
  author = {Peng Ye and Shengji Tang and Baopu Li and Tao Chen and Wanli Ouyang},
  title = {{Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure