Robust Knowledge Transfer via Hybrid Forward on the Teacher-Student Model

Liangchen Song, Jialian Wu, Ming Yang, Qian Zhang, Yuan Li, Junsong Yuan

Introduction

Transferring knowledge from a large, pre-trained "teacher" model to a smaller "student" model is a cornerstone of modern deep learning. However, this process often encounters significant friction when the teacher and student do not align in architecture, data domain, or specific task. This paper introduces Hybrid Forward, a novel training scheme that addresses these discrepancies by alternating sub-networks during the forward pass and employing a dynamic loss balancer to bridge gaps without manual hyperparameter tuning.

The Three Gaps in Knowledge Transfer

Standard knowledge distillation often fails due to three primary misalignments:

  1. Model Gap: The architectural difference between a heavy teacher (e.g., ResNet-101) and a lightweight student (e.g., MobileNet).
Model Gap: Transferring knowledge from a heavy ResNet-50 to a lightweight MobileNet v1.
  1. Domain Gap: Differences in data distribution, such as transferring from synthetic game frames (GTA5) to real-world street scenes (CityScapes).
  2. Task Gap: Discrepancies in objectives, where a teacher trained for one task (e.g., human pose estimation) is used to guide a student learning another (e.g., pedestrian detection).
Task Gap: Multi-task training where the teacher and student focus on different objectives like detection and segmentation.

Methodology

The proposed solution consists of two main components: Hybrid Forward and Dynamic Balancing.

Hybrid Forward

Instead of treating the teacher and student as isolated entities, the authors "stitch" their stages together. Both networks are divided into backbone stages. During training, data batches follow three distinct paths:

  • Path 1: Student Stage A $\rightarrow$ Teacher Stage B $\rightarrow$ Student Task Head.
  • Path 2: Teacher Stage A $\rightarrow$ Student Stage B $\rightarrow$ Student Task Head.
  • Path 3: Student Stage A $\rightarrow$ Student Stage B $\rightarrow$ Student Task Head (standard student path).

To resolve channel mismatches between different architectures, $1 \times 1$ convolutions act as adapters. Only the student weights and these adapters are updated; the teacher remains frozen.

Dynamic Balancing

To avoid the tedious process of manual loss weighting, the authors introduce a dynamic parameter, $\alpha$: $$\alpha = \frac{2 * L_3}{L_1 + L_2}$$ Here, $L_1, L_2, L_3$ are the task-specific losses from the three paths. This ratio ensures the transfer loss is scaled relative to the student's primary task loss. If the teacher's features are irrelevant to the new task, $\alpha$ decreases, preventing the teacher from negatively impacting the student's learning.

Experimental Results

The method was validated across several benchmarks:

  • ImageNet (Model Compression): A MobileNet-v1 student achieved 71.39% Top-1 accuracy, significantly outperforming standard distillation (68.58%) and previous state-of-the-art methods.
  • CityScapes (Domain/Task Gap): Transferring from a synthetic GTA5 segmentation teacher to a real-world student yielded an mIoU of 60.7.
  • CityPersons (Extreme Task Transfer): Using a pose estimation teacher (COCO) to help a pedestrian detection student reduced the miss-rate from 17.20 to 16.19, proving the method can extract useful feature-extraction capabilities even across wildly different tasks.

Conclusion

By forcing feature compatibility through hybrid paths and automating loss balancing, this approach makes transfer learning more robust and accessible across diverse architectures and tasks. While it increases training time due to multiple forward passes, the resulting student models maintain their original inference efficiency, making it a practical choice for deploying high-performance lightweight models.

Summary updated

Abstract

When adopting deep neural networks for a new vision task, a common practice is to start with fine-tuning some off-the-shelf well-trained network models from the community. Since a new task may require training a different network architecture with new domain data, taking advantage of off-the-shelf models is not trivial and generally requires considerable try-and-error and parameter tuning. In this paper, we denote a well-trained model as a teacher network and a model for the new task as a student network. We aim to ease the efforts of transferring knowledge from the teacher to the student network, robust to the gaps between their network architectures, domain data, and task definitions. Specifically, we propose a hybrid forward scheme in training the teacher-student models, alternately updating layer weights of the student model. The key merit of our hybrid forward scheme is on the dynamical balance between the knowledge transfer loss and task specific loss in training. We demonstrate the effectiveness of our method on a variety of tasks, e.g., model compression, segmentation, and detection, under a variety of knowledge transfer settings.
BibTeX
@inproceedings{song2021robust,
  author = {Liangchen Song and Jialian Wu and Ming Yang and Qian Zhang and Yuan Li and Junsong Yuan},
  title = {{Robust Knowledge Transfer via Hybrid Forward on the Teacher-Student Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i3.16358},
}
Figure