Introduction
Transferring knowledge from a large, pre-trained "teacher" model to a smaller "student" model is a cornerstone of modern deep learning. However, this process often encounters significant friction when the teacher and student do not align in architecture, data domain, or specific task. This paper introduces Hybrid Forward, a novel training scheme that addresses these discrepancies by alternating sub-networks during the forward pass and employing a dynamic loss balancer to bridge gaps without manual hyperparameter tuning.
The Three Gaps in Knowledge Transfer
Standard knowledge distillation often fails due to three primary misalignments:
- Model Gap: The architectural difference between a heavy teacher (e.g., ResNet-101) and a lightweight student (e.g., MobileNet).
- Domain Gap: Differences in data distribution, such as transferring from synthetic game frames (GTA5) to real-world street scenes (CityScapes).
- Task Gap: Discrepancies in objectives, where a teacher trained for one task (e.g., human pose estimation) is used to guide a student learning another (e.g., pedestrian detection).
Methodology
The proposed solution consists of two main components: Hybrid Forward and Dynamic Balancing.
Hybrid Forward
Instead of treating the teacher and student as isolated entities, the authors "stitch" their stages together. Both networks are divided into backbone stages. During training, data batches follow three distinct paths:
- Path 1: Student Stage A $\rightarrow$ Teacher Stage B $\rightarrow$ Student Task Head.
- Path 2: Teacher Stage A $\rightarrow$ Student Stage B $\rightarrow$ Student Task Head.
- Path 3: Student Stage A $\rightarrow$ Student Stage B $\rightarrow$ Student Task Head (standard student path).
To resolve channel mismatches between different architectures, $1 \times 1$ convolutions act as adapters. Only the student weights and these adapters are updated; the teacher remains frozen.
Dynamic Balancing
To avoid the tedious process of manual loss weighting, the authors introduce a dynamic parameter, $\alpha$: $$\alpha = \frac{2 * L_3}{L_1 + L_2}$$ Here, $L_1, L_2, L_3$ are the task-specific losses from the three paths. This ratio ensures the transfer loss is scaled relative to the student's primary task loss. If the teacher's features are irrelevant to the new task, $\alpha$ decreases, preventing the teacher from negatively impacting the student's learning.
Experimental Results
The method was validated across several benchmarks:
- ImageNet (Model Compression): A MobileNet-v1 student achieved 71.39% Top-1 accuracy, significantly outperforming standard distillation (68.58%) and previous state-of-the-art methods.
- CityScapes (Domain/Task Gap): Transferring from a synthetic GTA5 segmentation teacher to a real-world student yielded an mIoU of 60.7.
- CityPersons (Extreme Task Transfer): Using a pose estimation teacher (COCO) to help a pedestrian detection student reduced the miss-rate from 17.20 to 16.19, proving the method can extract useful feature-extraction capabilities even across wildly different tasks.
Conclusion
By forcing feature compatibility through hybrid paths and automating loss balancing, this approach makes transfer learning more robust and accessible across diverse architectures and tasks. While it increases training time due to multiple forward passes, the resulting student models maintain their original inference efficiency, making it a practical choice for deploying high-performance lightweight models.
Summary updated