Tree-structured Auxiliary Online Knowledge Distillation

Wenye Lin, Yangning Li, Yifeng Ding, Hai-Tao Zheng

Introduction

Knowledge Distillation (KD) has long been the standard technique for compressing large, "cumbersome" neural networks into compact, efficient ones. However, traditional KD is a two-stage process requiring a pre-trained "teacher" model. Online Knowledge Distillation (OKD) offers a more efficient alternative by training multiple student models (peers) simultaneously in a single stage, allowing them to learn from each other.

While previous OKD research has focused heavily on designing complex distillation objectives—such as attention mechanisms or gating systems to weight peer feedback—this paper argues that the global architecture of the training setup is a more powerful, yet overlooked, lever for performance. The authors introduce Tree-structured Auxiliary online knowledge distillation (TSA), which shifts the focus from complex loss functions to architectural design by arranging auxiliary branches into a hierarchical tree structure.

Methodology: The Tree-Structured Architecture

TSA transforms a single target network into a tree-like structure during training. Instead of training identical models side-by-side, the approach splits the network into hierarchical blocks.

  1. The Tree Structure: Parallel "counterpart" branches are added to the later layers of the network. This creates a structure where the early layers (the "trunk") are shared, while the later layers (the "branches") provide multiple independent paths to a final classification.
  2. General to Specific Features: This design mimics the hierarchical nature of deep networks, where early layers extract general features (edges, textures) and later layers extract task-specific features. By branching out later, the model generates multiple "opinions" on the specific task while grounding them in the same foundational features.
  3. Simplified Distillation: TSA utilizes the simplest possible distillation loss—standard KL Divergence—without additional complexity. This demonstrates that the architectural arrangement itself, rather than a sophisticated loss function, is the primary driver of performance gains.
Comparison between a standard sequential layer stack and the proposed TSA tree-structured architecture.

After training is complete, all auxiliary branches are discarded. The final model retains the exact parameter count and inference speed of the original "vanilla" architecture but benefits from the enhanced knowledge acquired during the multi-branch training phase.

Experimental Evaluation

The researchers evaluated TSA across diverse domains, including Computer Vision (CIFAR-10, CIFAR-100, and ImageNet-1K) and Natural Language Processing (IWSLT and WMT translation tasks). This study marks the first successful demonstration of online knowledge distillation significantly boosting performance in Machine Translation using the Transformer architecture.

Key Results and Insights

  • State-of-the-Art Performance: TSA achieved SOTA results among online KD methods. On CIFAR-100, models like MobileNetV3 and ResNet-18 reached their highest reported accuracies. On ImageNet, a ResNet-18 trained with TSA achieved 81.43% accuracy, actually outperforming a vanilla ResNet-101.
  • The Regularization Effect: TSA acts as a powerful regularizer. During training, the model often shows lower training accuracy but significantly higher test accuracy compared to vanilla models, suggesting superior generalization and reduced overfitting.
  • Translation Gains: In NMT tasks, TSA provided a stable gain of approximately 0.9 BLEU points over the vanilla Transformer, with a much smoother and more stable training curve.
Training and test accuracy curves for ResNet-18 with and without TSA, highlighting the regularization effect.

Discussion and Future Work

TSA proves that architecture can be more impactful than loss function design in online distillation. By providing different "views" of the input through different branch initializations, the model naturally encourages diversity.

However, the method does involve a training overhead; while inference is "free," training requires more GPU memory and time to accommodate the auxiliary branches. Future work may explore combining TSA's tree architecture with more advanced distillation losses or investigating more complex, non-binary tree structures to further push the boundaries of model performance.

Summary updated

BibTeX
@inproceedings{lin2022tree,
  author = {Wenye Lin and Yangning Li and Yifeng Ding and Hai-Tao Zheng},
  title = {{Tree-structured Auxiliary Online Knowledge Distillation}},
  booktitle = {IJCNN},
  year = {2022},
  doi = {10.1109/ijcnn55064.2022.9892764},
}
Figure