Abstract
Residual Networks (ResNets) are often interpreted as an implicit ensemble of numerous sub-networks. This paper introduces Group Knowledge based Training (GKT), a framework that organizes these subnets into hierarchical groups. By leveraging knowledge from larger, more capable groups to supervise smaller ones, GKT reduces the capacity gap between teacher and student. This hierarchical mentoring leads to significant accuracy improvements and reduced training times across various architectures, including CNNs and Transformers.
The Unraveled View and the Capacity Gap
Residual Networks are the backbone of modern deep learning, yet the mechanisms behind their success are still being explored. A prominent theory, the "unraveled view," posits that a ResNet with $L$ residual blocks is effectively an ensemble of $2^L$ subnets of varying lengths. While previous methods like Stochastic Depth or Stimulative Training attempt to improve performance by training these subnets, they often encounter a capacity gap. These methods typically provide the same level of supervision—either ground-truth labels or the complex output of the full "main" network—to every subnet regardless of its size. Tiny subnets frequently lack the capacity to mimic the full network perfectly, leading to suboptimal learning and "network loafing."
The GKT Framework
To address the capacity gap, the authors propose the GKT framework. GKT organizes subnets into hierarchical groups based on their size and uses knowledge from larger groups to supervise smaller ones, creating a "teacher assistant" effect where supervision is tailored to the student's ability.
The framework consists of four key components:
- Subnet-in-Subnet (SIS) Sampling: Subnets are sampled from "parent" subnets, naturally creating a hierarchy (e.g., Large, Medium, Tiny).
- Group Knowledge Aggregation: Each group maintains an Exponential Moving Average (EMA) of its subnets' logits, providing a stable, collective target for supervision.
- Hierarchical Knowledge Transfer: Larger groups supervise smaller ones, tailoring the difficulty of the learning task to the student's capacity.
- Inheriting Exponential Decay Rule: A sampling bias favors larger subnets, which have been found to benefit the final main network more significantly.
Empirical Evaluation
GKT was evaluated on standard benchmarks including CIFAR-10/100, Tiny ImageNet, and ImageNet-1K, using architectures ranging from ResNets and MobileNetV3 to Vision Transformers (ViT, Swin). Results consistently showed that GKT outperforms standard training and existing self-distillation methods.
Key performance highlights include:
- Accuracy Gains: ResNet-50 saw a 3.59% improvement on CIFAR-100 and a 3.68% boost on Tiny ImageNet. ViT-S improved by 4.57% on Tiny ImageNet.
- Efficiency: On ImageNet-1K, GKT reduced ResNet-50 training time by approximately 43% (from 345 to 194 hours) while achieving superior accuracy compared to Stimulative Training.
- Generalization: Representations learned via GKT transferred effectively to downstream tasks like object detection and segmentation on COCO2017.
Discussion and Insights
The success of GKT stems from its "sociological" approach to knowledge transfer, where information flows more effectively between groups of similar capacity. By bridging the "sharpness gap"—a metric for the capacity difference—GKT makes the learning task more manageable for smaller subnets. Furthermore, because it utilizes subnets already present in the architecture, GKT avoids the overhead of separate teacher models, making efficiency a core feature rather than an afterthought.
Conclusion
GKT provides a robust and efficient framework for boosting the performance of residual-based architectures. While it introduces some sampling complexity and new hyperparameters, the significant gains in accuracy and training speed make it a compelling alternative to standard training. Future work may explore its application to Large Language Models (LLMs), which also rely heavily on residual connections.
Summary updated