Abstract
In multi-task learning (MTL) for visual scene understanding, it is crucial to transfer useful information between multiple tasks with minimal interferences. In this paper, we propose a novel architecture that effectively transfers informative features by applying the attention mechanism to the multi-scale features of the tasks. Since applying the attention module directly to all possible features in terms of scale and task requires a high complexity, we propose to apply the attention module sequentially for the task and scale. The cross-task attention module (CTAM) is first applied to facilitate the exchange of relevant information between the multiple task features of the same scale. The cross-scale attention module (CSAM) then aggregates useful information from feature maps at different resolutions in the same task. Also, we attempt to capture long range dependencies through the self-attention module in the feature extraction network. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the NYUD-v2 and PASCAL-Context dataset.
Introduction
In the field of computer vision, scene understanding is a multi-faceted challenge. A comprehensive understanding of a visual scene requires an AI to simultaneously perform semantic segmentation (identifying objects), depth estimation (calculating distance), and surface normal estimation (determining surface orientation). While training independent models for each task is possible, Multi-Task Learning (MTL) offers a more efficient alternative by sharing information across tasks.
However, MTL faces a significant hurdle: task interference. If information is shared indiscriminately, the performance of individual tasks can degrade. Furthermore, relating every feature at every resolution across every task creates an exponential increase in computational cost. This paper addresses the "Goldilocks" problem of MTL—finding the optimal balance of information sharing across both tasks and scales without incurring prohibitive computational overhead.
Methodology
The core innovation of this research is the Sequential Cross Attention (SCA) mechanism. Rather than calculating relationships between all possible feature pairs simultaneously, the architecture decouples task-to-task and scale-to-scale interactions into sequential modules.
1. Cross-Task Attention Module (CTAM)
CTAM facilitates the exchange of information between different tasks at the same resolution scale. For instance, it allows depth cues to inform the semantic segmentation task. By focusing on task interactions within a single scale, the module acts as a smart filter, allowing a target task to select only the most relevant features from source tasks.
2. Cross-Scale Attention Module (CSAM)
Following the task-specific refinement in CTAM, the CSAM aggregates information from different resolutions within the same task. This ensures that fine-grained local details (crucial for depth) and global context (crucial for segmentation) are effectively integrated.
3. Self-Attention Backbone and FPMA
Unlike traditional MTL models that rely on standard Convolutional Neural Networks (CNNs), this architecture incorporates self-attention blocks—inspired by the Swin Transformer—directly into the backbone. This enables the model to capture long-range dependencies and global context early in the feature extraction process. Additionally, a Feature Propagation Module with Attention (FPMA) ensures that low-resolution information with a large receptive field is effectively passed to higher-resolution layers.
Experiments and Results
The model, designated SCA-MTL, was evaluated on two primary benchmarks: NYUD-v2 and PASCAL-Context.
NYUD-v2 Performance
On the NYUD-v2 dataset (covering semantic segmentation, depth estimation, and surface normal estimation), SCA-MTL achieved a +4.82% improvement in overall multi-task performance ($\Delta_m$) relative to a single-task baseline. Notably, it established new performance highs for semantic segmentation (40.50 mIoU) and depth estimation (0.584 RMSE).
PASCAL-Context Performance
In the PASCAL-Context challenge, the model demonstrated a +5.69% improvement over the baseline. This significantly outperformed previous state-of-the-art methods, such as ATRC, which achieved a +2.73% improvement.
Ablation Studies
Ablation experiments confirmed that each component—self-attention, FPMA, CTAM, and CSAM—contributed incrementally to the final performance. The sequential application of task and scale attention proved superior to simultaneous or dense attention approaches in both accuracy and efficiency.
Conclusion
The SCA-MTL architecture demonstrates that sequential cross-attention is a highly effective strategy for multi-task scene understanding. By decoupling task and scale interactions, the model achieves state-of-the-art results while maintaining manageable computational complexity. Future work may involve utilizing Neural Architecture Search (NAS) to automatically optimize the arrangement of these attention modules, potentially further enhancing performance across diverse visual tasks.
Summary updated