Introduction
Multi-task learning (MTL) is a paradigm in computer vision aimed at solving multiple dense prediction tasks—such as semantic segmentation, depth estimation, and edge detection—within a single unified framework. While MTL promises increased efficiency and potential performance gains through cross-task synergy, it often faces the challenge of "negative transfer," where learning one task inadvertently degrades the performance of another. MTI-Net (Multi-Scale Task Interaction Network) addresses this by explicitly modeling task interactions across multiple spatial scales.
The Problem: Scale-Agnostic Interaction
Traditional MTL architectures often perform task distillation at a single, fixed resolution. However, the relationship between tasks is inherently scale-dependent. For instance, at a local level, depth discontinuities are highly correlated with semantic edges. Conversely, at a global scale, the overall structure of an object in a depth map provides a more robust cue for its semantic category than localized edges do. Failing to account for these varying affinities leads to suboptimal feature sharing and contributes to negative transfer.
Methodology
MTI-Net introduces a multi-scale approach to task interaction, designed to be backbone-agnostic. The architecture is built upon three core pillars:
- Multi-Scale Multi-Modal Distillation: The network extracts task-specific features at four distinct resolutions (1/4, 1/8, 1/16, and 1/32). At each scale, a spatial attention mechanism allows tasks to selectively "borrow" relevant information from one another.
- Feature Propagation Module (FPM): To ensure global context informs local details, features are propagated from coarser (lower resolution) scales to finer (higher resolution) scales. This allows high-resolution layers to benefit from the wider receptive fields of deeper layers.
- Feature Aggregation: Refined features from all scales are upsampled and aggregated to produce the final predictions for each task.
Key Insights: Scale-Dependent Affinity
A central contribution of this work is the quantitative demonstration that task affinities fluctuate with scale. By analyzing pixel affinities across different tasks at varying dilations, the authors confirmed that correlations (e.g., between semantics and edges) diminish as the spatial context expands. This finding validates the necessity of the multi-scale distillation approach.
Furthermore, MTI-Net employs "deep supervision" by generating initial task predictions at every resolution. This forces intermediate layers to develop task-specific representations early in the network, rather than relying solely on generic backbone features.
Experimental Results
The model was evaluated on the NYUD-v2 and PASCAL-Context benchmarks, demonstrating superior performance and efficiency:
- NYUD-v2: MTI-Net achieved a +10.91% relative improvement over single-task baselines, outperforming state-of-the-art MTL models like PAD-Net and ASTMT.
- PASCAL-Context: The network surpassed single-task models by +1.36% in average performance while being significantly more efficient, utilizing 35% fewer parameters and 19.9% fewer FLOPs.
Conclusion
MTI-Net effectively mitigates negative transfer in multi-task learning by recognizing that task interactions are not uniform across scales. By distilling and propagating features across multiple resolutions, it achieves a state-of-the-art balance between accuracy and computational efficiency. While some challenges remain—such as handling noisy labels in specific tasks like surface normals—MTI-Net provides a robust foundation for future research into complex, multi-modal scene understanding.
Summary updated