Introduction
Residual networks (ResNets) are foundational in modern deep learning, often interpreted through the "unraveled view" as implicit ensembles of many shallower subnets. While techniques like Stochastic Depth and Stimulative Training (ST) attempt to optimize these subnets individually, they frequently encounter a "capacity gap." Smaller subnets often lack the representational power to mimic the complex outputs of the full "Main Net," leading to suboptimal learning. Furthermore, standard training can result in "network loafing," where certain layers fail to contribute effectively to the ensemble.
To address these challenges, we introduce Group Knowledge based Training (GKT). GKT organizes subnets into hierarchical groups based on their capacity and utilizes the aggregated knowledge of larger subnets to provide tailored, digestible supervision for smaller ones. This hierarchical approach bridges the distillation gap and improves training efficiency across both Convolutional Neural Networks (CNNs) and Vision Transformers.
Methodology
The GKT framework is built upon three primary components designed to facilitate efficient knowledge transfer within the implicit ensemble of a residual network.
Subnet-in-Subnet (SIS) Sampling and Grouping
GKT employs a hierarchical sampling strategy called Subnet-in-Subnet (SIS) sampling. Instead of uniform random sampling, it selects a "parent" subnet and subsequently samples "child" subnets from within it. This process naturally categorizes subnets into hierarchical groups—Tiny, Median, and Large—based on their inclusion relationships and computational capacity.
Group Knowledge Aggregation
To provide stable supervision without the overhead of multiple teacher models, GKT maintains an Exponential Moving Average (EMA) of output logits for subnets within each group. This "Group Knowledge" represents the collective wisdom of the group's members. Because logits are updated during standard training passes, the additional computational cost is minimal.
Hierarchical Knowledge Transfer
The core innovation of GKT is its hierarchical supervision. Rather than the Main Net directly supervising all subnets, the Large Subnet Group acts as a teacher for the Medium Group, which in turn supervises the Tiny Group. This step-by-step transfer ensures that the "knowledge" is appropriate for the target subnet's capacity, making it easier to absorb. Additionally, an Inheriting Exponential Decay Rule is used to prioritize the training of larger subnets, which have a more significant impact on the final model's performance.
Experiments
The effectiveness of GKT was evaluated across several benchmarks and architectures:
- Image Classification: CIFAR-100, TinyImageNet, and ImageNet-1K.
- Downstream Tasks: Object detection and segmentation on COCO2017 using Faster R-CNN and Mask R-CNN.
- Architectures: CNNs including ResNet-34/50, WideResNet (WRN), and MobileNetV3, as well as Vision Transformers such as ViT, Swin, and CaiT.
Results
GKT demonstrates significant performance improvements over standard training and existing methods like Stimulative Training (ST). On CIFAR-100, GKT improved Top-1 accuracy by up to 1.67%. The gains were even more pronounced for Transformers, with ViT-S seeing a 4.57% accuracy boost on TinyImageNet.
Crucially, GKT is highly efficient. As shown in the comparative analysis, GKT often achieves superior accuracy compared to ST while requiring significantly fewer GPU hours. This efficiency stems from GKT's ability to provide high-quality supervision without always requiring a full forward pass of the most computationally expensive "Main Net." Furthermore, GKT-trained models exhibit a lower "sharpness gap," suggesting better generalization capabilities.
Conclusion
Group Knowledge based Training (GKT) offers a robust framework for optimizing residual-based architectures by treating them as hierarchical ensembles. By bridging the capacity gap through intermediate group-level supervision, GKT achieves state-of-the-art performance with high training efficiency. While the method introduces new hyperparameters like the EMA coefficient, its universal applicability to both CNNs and Transformers highlights its potential as a fundamental training paradigm for modern deep networks.
Summary updated