MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning

Simon Vandenhende, Stamatios Georgoulis, Luc Van GoolarXiv 2020 · arXiv

Introduction

Multi-task learning (MTL) is a paradigm in computer vision aimed at solving multiple dense prediction tasks—such as semantic segmentation, depth estimation, and edge detection—within a single unified framework. While MTL promises increased efficiency and potential performance gains through cross-task synergy, it often faces the challenge of "negative transfer," where learning one task inadvertently degrades the performance of another. MTI-Net (Multi-Scale Task Interaction Network) addresses this by explicitly modeling task interactions across multiple spatial scales.

The Problem: Scale-Agnostic Interaction

Traditional MTL architectures often perform task distillation at a single, fixed resolution. However, the relationship between tasks is inherently scale-dependent. For instance, at a local level, depth discontinuities are highly correlated with semantic edges. Conversely, at a global scale, the overall structure of an object in a depth map provides a more robust cue for its semantic category than localized edges do. Failing to account for these varying affinities leads to suboptimal feature sharing and contributes to negative transfer.

Methodology

MTI-Net introduces a multi-scale approach to task interaction, designed to be backbone-agnostic. The architecture is built upon three core pillars:

  1. Multi-Scale Multi-Modal Distillation: The network extracts task-specific features at four distinct resolutions (1/4, 1/8, 1/16, and 1/32). At each scale, a spatial attention mechanism allows tasks to selectively "borrow" relevant information from one another.
  2. Feature Propagation Module (FPM): To ensure global context informs local details, features are propagated from coarser (lower resolution) scales to finer (higher resolution) scales. This allows high-resolution layers to benefit from the wider receptive fields of deeper layers.
  3. Feature Aggregation: Refined features from all scales are upsampled and aggregated to produce the final predictions for each task.
The MTI-Net architecture featuring multi-scale distillation and feature propagation modules.

Key Insights: Scale-Dependent Affinity

A central contribution of this work is the quantitative demonstration that task affinities fluctuate with scale. By analyzing pixel affinities across different tasks at varying dilations, the authors confirmed that correlations (e.g., between semantics and edges) diminish as the spatial context expands. This finding validates the necessity of the multi-scale distillation approach.

Quantitative proof that task affinities, such as Semantics/Edges and Semantics/Depth, vary significantly across different spatial scales.

Furthermore, MTI-Net employs "deep supervision" by generating initial task predictions at every resolution. This forces intermediate layers to develop task-specific representations early in the network, rather than relying solely on generic backbone features.

Experimental Results

The model was evaluated on the NYUD-v2 and PASCAL-Context benchmarks, demonstrating superior performance and efficiency:

  • NYUD-v2: MTI-Net achieved a +10.91% relative improvement over single-task baselines, outperforming state-of-the-art MTL models like PAD-Net and ASTMT.
  • PASCAL-Context: The network surpassed single-task models by +1.36% in average performance while being significantly more efficient, utilizing 35% fewer parameters and 19.9% fewer FLOPs.

Conclusion

MTI-Net effectively mitigates negative transfer in multi-task learning by recognizing that task interactions are not uniform across scales. By distilling and propagating features across multiple resolutions, it achieves a state-of-the-art balance between accuracy and computational efficiency. While some challenges remain—such as handling noisy labels in specific tasks like surface normals—MTI-Net provides a robust foundation for future research into complex, multi-modal scene understanding.

Summary updated

Abstract

In this paper, we argue about the importance of considering task interactions at multiple scales when distilling task information in a multi-task learning setup. In contrast to common belief, we show that tasks with high affinity at a certain scale are not guaranteed to retain this behaviour at other scales, and vice versa. We propose a novel architecture, namely MTI-Net, that builds upon this finding in three ways. First, it explicitly models task interactions at every scale via a multi-scale multi-modal distillation unit. Second, it propagates distilled task information from lower to higher scales via a feature propagation module. Third, it aggregates the refined task features from all scales via a feature aggregation unit to produce the final per-task predictions. Extensive experiments on two multi-task dense labeling datasets show that, unlike prior work, our multi-task model delivers on the full potential of multi-task learning, that is, smaller memory footprint, reduced number of calculations, and better performance w.r.t. single-task learning. The code is made publicly available: https://github.com/SimonVandenhende/Multi-Task-Learning-PyTorch.
BibTeX
@misc{vandenhende2020mti,
  author = {Simon Vandenhende and Stamatios Georgoulis and Luc Van Gool},
  title = {{MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning}},
  year = {2020},
  eprint = {2001.06902},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.06902},
}
Figure