Online Knowledge Distillation for Multi-Task Learning

Geethu Miriam Jacob, Vishal Agarwal, Björn StengerWACV 2023

TL;DR

This paper introduces a novel method for multi-task learning (MTL) that simultaneously trains single-task models and a multi-task model. By using an "online knowledge distillation" approach, the single-task models guide the multi-task model's learning process, effectively mitigating "negative transfer" and achieving performance comparable to individual single-task models while maintaining the efficiency of MTL.

What Problem Does This Paper Address?

Multi-task learning (MTL) is a powerful paradigm for training a single model to perform multiple tasks simultaneously, offering benefits like memory efficiency and faster inference. However, a significant challenge in MTL is "negative transfer," where training for one task can inadvertently degrade performance on another. This often happens because a single shared representation might not be optimal for all tasks. Existing methods to combat negative transfer include task weighting, feature fusion, and knowledge distillation, but these often rely on pre-trained models or complex multi-stage training. This paper aims to address negative transfer by proposing a more integrated and efficient approach: simultaneously training individual single-task models alongside the multi-task model, using the former to actively guide the latter during training.

Methods

The core of this paper's contribution lies in its "online knowledge distillation" framework. Instead of relying on pre-trained "teacher" models, it trains multiple single-task (STL) networks concurrently with the main multi-task (MTL) network. This simultaneous training allows the STL networks to act as dynamic guides for the MTL network.

Two key components enable this:

  • Adaptive Feature Distillation (AFD): This technique selectively distills knowledge from intermediate features of the STL networks to the MTL network. The "adaptive" aspect comes from learning weights for each layer and task, allowing the MTL network to learn which features are most important from each STL model. Crucially, gradients from the distillation loss are not back-propagated to the STL networks to prevent them from being negatively influenced by the MTL parameters.
  • Online Task Weighting (OTW): This scheme dynamically adjusts the contribution of each task's loss to the overall MTL training objective. The weights are calculated based on the ratio of the MTL loss to the STL loss for each task at each training iteration. Tasks with a larger discrepancy between their MTL and STL losses receive higher weights, encouraging the MTL model to catch up.

The overall loss function combines the AFD loss with the individual task losses for both STL and MTL networks, weighted by the OTW scheme.

Datasets

The proposed method is evaluated on two widely used benchmark datasets for scene understanding tasks:

  • NYUv2: This dataset contains RGB-D images from indoor scenes and is used for evaluating semantic segmentation, depth estimation, and surface normal estimation.
  • Cityscapes: This dataset features high-resolution street-view images and is used for evaluating semantic segmentation and depth estimation.

The authors used pre-processed versions of these datasets provided by the MTAN [20] codebase.

Key Insights and Ideas

  • Simultaneous Training as a Teacher-Student Relationship: The paper elegantly frames the STL networks as dynamic "teachers" for the MTL "student" network, but crucially, this relationship is established during training, not before. This allows for continuous adaptation and guidance.
  • Adaptive Feature Distillation is Key: The insight that different layers contribute differently to various tasks is central to AFD. By adaptively distilling features, the MTL network can learn a more nuanced and task-specific representation from the shared backbone.
  • Online Task Weighting Balances the Scales: The OTW mechanism is vital for ensuring that no single task dominates or is neglected. By dynamically adjusting weights based on current performance discrepancies, it helps to prevent negative transfer and promote balanced learning across all tasks.
  • ViT Backbone Compatibility: The framework is designed to be backbone-agnostic, but the authors demonstrate its effectiveness with Vision Transformers (ViTs), a modern and powerful architecture, highlighting its adaptability to current state-of-the-art models.

Key Results

The proposed method, "OKD-MTL," demonstrates significant improvements over baseline MTL methods on both datasets:

  • NYUv2: Achieved a 6.22% improvement over the baseline MTL model, outperforming the next best method (KD-MTL) by 1.3%.
  • Cityscapes: Showed a 9.19% improvement over the baseline, with the Random Loss Weighting (RLW) method being the closest competitor at 7.69%.

Ablation studies confirmed the effectiveness of both AFD and OTW components, with their combination yielding the best results. The paper also notes that while the MTL model uses more parameters than a single-task model, it's significantly more efficient than running multiple separate STL models, offering a 57% size reduction and a 60% inference time reduction compared to separate models.

Limitations and Future Work

  • Computational Cost of Simultaneous Training: While the inference is efficient, the simultaneous training of multiple STL networks alongside the MTL network requires more computational resources during the training phase compared to training a single MTL model. The paper acknowledges this trade-off.
  • Hyperparameter Sensitivity: The OTW scheme includes a temperature parameter (T) that influences the distribution of task weights. While T=0.1 was found to be optimal for the NYU dataset, further investigation into its sensitivity and optimal selection across different datasets and task configurations could be beneficial.
  • Extending to More Complex Scenarios: The current work focuses on pixel-wise prediction tasks. Future work could explore extending this online distillation framework to other types of tasks and more complex multi-task learning scenarios.

Summary updated

BibTeX
@inproceedings{jacob2023online,
  author = {Geethu Miriam Jacob and Vishal Agarwal and Björn Stenger},
  title = {{Online Knowledge Distillation for Multi-Task Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure