Boosting Residual Networks with Group Knowledge

Shengji Tang, Peng Ye, Baopu Li, Weihao Lin, Tao Chen, Tong He, Chong Yu, Wanli Ouyang

Introduction

Residual networks (ResNets) are foundational in modern deep learning, often interpreted through the "unraveled view" as implicit ensembles of many shallower subnets. While techniques like Stochastic Depth and Stimulative Training (ST) attempt to optimize these subnets individually, they frequently encounter a "capacity gap." Smaller subnets often lack the representational power to mimic the complex outputs of the full "Main Net," leading to suboptimal learning. Furthermore, standard training can result in "network loafing," where certain layers fail to contribute effectively to the ensemble.

To address these challenges, we introduce Group Knowledge based Training (GKT). GKT organizes subnets into hierarchical groups based on their capacity and utilizes the aggregated knowledge of larger subnets to provide tailored, digestible supervision for smaller ones. This hierarchical approach bridges the distillation gap and improves training efficiency across both Convolutional Neural Networks (CNNs) and Vision Transformers.

Methodology

The GKT framework is built upon three primary components designed to facilitate efficient knowledge transfer within the implicit ensemble of a residual network.

Subnet-in-Subnet (SIS) Sampling and Grouping

GKT employs a hierarchical sampling strategy called Subnet-in-Subnet (SIS) sampling. Instead of uniform random sampling, it selects a "parent" subnet and subsequently samples "child" subnets from within it. This process naturally categorizes subnets into hierarchical groups—Tiny, Median, and Large—based on their inclusion relationships and computational capacity.

The unraveled view of ResNets and the GKT framework for dividing subnets into groups and aggregating knowledge.

Group Knowledge Aggregation

To provide stable supervision without the overhead of multiple teacher models, GKT maintains an Exponential Moving Average (EMA) of output logits for subnets within each group. This "Group Knowledge" represents the collective wisdom of the group's members. Because logits are updated during standard training passes, the additional computational cost is minimal.

Hierarchical Knowledge Transfer

The core innovation of GKT is its hierarchical supervision. Rather than the Main Net directly supervising all subnets, the Large Subnet Group acts as a teacher for the Medium Group, which in turn supervises the Tiny Group. This step-by-step transfer ensures that the "knowledge" is appropriate for the target subnet's capacity, making it easier to absorb. Additionally, an Inheriting Exponential Decay Rule is used to prioritize the training of larger subnets, which have a more significant impact on the final model's performance.

Experiments

The effectiveness of GKT was evaluated across several benchmarks and architectures:

  • Image Classification: CIFAR-100, TinyImageNet, and ImageNet-1K.
  • Downstream Tasks: Object detection and segmentation on COCO2017 using Faster R-CNN and Mask R-CNN.
  • Architectures: CNNs including ResNet-34/50, WideResNet (WRN), and MobileNetV3, as well as Vision Transformers such as ViT, Swin, and CaiT.

Results

GKT demonstrates significant performance improvements over standard training and existing methods like Stimulative Training (ST). On CIFAR-100, GKT improved Top-1 accuracy by up to 1.67%. The gains were even more pronounced for Transformers, with ViT-S seeing a 4.57% accuracy boost on TinyImageNet.

Comparison of training time (GPU hours) and Top-1 accuracy across various architectures for GKT, Baseline, and Stimulative Training (ST).

Crucially, GKT is highly efficient. As shown in the comparative analysis, GKT often achieves superior accuracy compared to ST while requiring significantly fewer GPU hours. This efficiency stems from GKT's ability to provide high-quality supervision without always requiring a full forward pass of the most computationally expensive "Main Net." Furthermore, GKT-trained models exhibit a lower "sharpness gap," suggesting better generalization capabilities.

Conclusion

Group Knowledge based Training (GKT) offers a robust framework for optimizing residual-based architectures by treating them as hierarchical ensembles. By bridging the capacity gap through intermediate group-level supervision, GKT achieves state-of-the-art performance with high training efficiency. While the method introduces new hyperparameters like the EMA coefficient, its universal applicability to both CNNs and Transformers highlights its potential as a fundamental training paradigm for modern deep networks.

Summary updated

Abstract

Recent research understands the residual networks from a new perspective of the implicit ensemble model. From this view, previous methods such as stochastic depth and stimulative training have further improved the performance of the residual network by sampling and training of its subnets. However, they both use the same supervision for all subnets of different capacities and neglect the valuable knowledge generated by subnets during training. In this manuscript, we mitigate the significant knowledge distillation gap caused by using the same kind of supervision and advocate leveraging the subnets to provide diverse knowledge. Based on this motivation, we propose a group knowledge based training framework for boosting the performance of residual networks. Specifically, we implicitly divide all subnets into hierarchical groups by subnet-in-subnet sampling, aggregate the knowledge of different subnets in each group during training, and exploit upper-level group knowledge to supervise lower-level subnet group. Meanwhile, we also develop a subnet sampling strategy that naturally samples larger subnets, which are found to be more helpful than smaller subnets in boosting performance for hierarchical groups. Compared with typical subnet training and other methods, our method achieves the best efficiency and performance trade-offs on multiple datasets and network structures. The code is at https://github.com/tsj-001/AAAI24-GKT.
BibTeX
@inproceedings{tang2024boosting,
  author = {Shengji Tang and Peng Ye and Baopu Li and Weihao Lin and Tao Chen and Tong He and Chong Yu and Wanli Ouyang},
  title = {{Boosting Residual Networks with Group Knowledge}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28322},
}
Figure