Sequential Cross Attention Based Multi-Task Learning

Sunkyung Kim, Hyesong Choi, Dongbo Min

In the world of computer vision, "scene understanding" isn't just one job—it’s a bundle of them. To truly understand a photo, an AI needs to know what objects are there (semantic segmentation), how far away they are (depth estimation), and which way surfaces are facing (surface normal estimation). While we could train three separate models, Multi-Task Learning (MTL) aims to do it all at once, sharing information between tasks to make the whole system more efficient and accurate.

However, sharing isn't always easy. If tasks interfere with each other, performance drops. A new paper from researchers at Ewha Womans University, titled "Sequential Cross Attention Based Multi-task Learning," proposes a clever way to let tasks "talk" to each other across different scales without breaking the computational bank.

TL;DR

The researchers developed a novel architecture that uses sequential cross-attention to share information across different tasks and different feature resolutions. By decoupling task-to-task and scale-to-scale interactions, they achieved state-of-the-art performance on major benchmarks like NYUD-v2 and PASCAL-Context while keeping computational complexity manageable.

What Problem Does This Paper Address?

Multi-task learning faces a "Goldilocks" problem. If you share too little information between tasks, you lose the benefit of joint learning. If you try to share everything—relating every feature at every resolution across every task—the computational cost explodes.

Previous methods often only looked at interactions between tasks at the same resolution. But different tasks thrive at different scales: depth estimation might need fine-grained local details, while semantic segmentation benefits from a broader, global context. The challenge is: How do we effectively transfer informative features across both tasks and scales without the math becoming prohibitively expensive?

Methods

The core innovation is the Sequential Cross Attention mechanism. Instead of trying to calculate the relationship between every possible pair of features at once, the authors break it down into two specialized modules:

  1. Cross-Task Attention Module (CTAM): This module facilitates the exchange of information between different tasks (e.g., sharing depth cues with the segmentation task) but keeps them at the same resolution scale.
  2. Cross-Scale Attention Module (CSAM): Once the tasks have "talked" to each other, this module refines the features by aggregating information from different resolutions within the same task.
  3. Self-Attention Backbone: Unlike traditional MTL models that rely solely on standard CNNs, this architecture injects self-attention (inspired by the Swin Transformer) directly into the backbone. This helps the model capture long-range dependencies—essentially "seeing the big picture"—from the very start.
  4. Feature Propagation Module with Attention (FPMA): This ensures that useful low-resolution information (which has a larger receptive field) is effectively passed up to higher-resolution layers.

By applying these modules sequentially, the model gains the benefits of multi-scale, multi-task interaction at a fraction of the cost of a "dense" all-to-all attention approach.

Datasets

The model was evaluated on two heavy-hitters in the computer vision world:

  • NYUD-v2: A dataset of indoor scenes used for three tasks: monocular depth estimation, semantic segmentation, and surface normal estimation.
  • PASCAL-Context: A complex dataset used for semantic segmentation, part segmentation, and surface normal estimation.

Key Insights and Ideas

  • Sequential is Better than Simultaneous: The "aha moment" here is that you don't need to correlate everything at once. By handling task-interaction first and scale-interaction second, the model achieves better results with lower complexity.
  • Attention as a "Filter": Rather than just concatenating features (the old-school way), the Cross-Attention Module (CAM) acts as a smart filter, allowing a "target" task to pick and choose exactly which "source" features from other tasks are actually helpful.
  • Global Context Matters Early: Adding self-attention to the backbone (the "ATT" blocks in their diagrams) provided a significant performance boost, proving that even dense prediction tasks benefit from global context early in the feature extraction process.

Key Results

The proposed model, nicknamed SCA-MTL, outperformed several recent high-performance models (like PADNet, MTINet, and ATRC).

  • NYUD-v2 (3 Tasks): The model achieved a +4.82% improvement in overall multi-task performance ($\Delta_m$) compared to a single-task baseline. It specifically hit a new high for semantic segmentation (40.50 mIoU) and depth estimation (0.584 RMSE).
  • PASCAL-Context: On the two-task challenge, it showed a +5.69% improvement over the baseline, significantly outperforming the previous state-of-the-art (ATRC) which sat at +2.73%.
  • Ablation Success: Their ablation study (Table 5) showed that every single component—the self-attention, the FPMA, the CTAM, and the CSAM—contributed to the final score, with the full "sequential" stack providing the highest gains.

Limitations and Future Work

While the paper demonstrates impressive accuracy, it doesn't dive deeply into the specific inference latency (frames per second) compared to simpler models, though it notes the complexity is theoretically lower than dense attention.

The authors suggest that Neural Architecture Search (NAS)—using AI to design the network structure itself—could be the next step. They believe that automatically finding the optimal way to arrange these attention modules could push performance even further.

Check out the code: The researchers have made their implementation available on GitHub: https://github.com/kimsunkyung/SCA-MTL

Summary updated

Abstract

In multi-task learning (MTL) for visual scene understanding, it is crucial to transfer useful information between multiple tasks with minimal interferences. In this paper, we propose a novel architecture that effectively transfers informative features by applying the attention mechanism to the multi-scale features of the tasks. Since applying the attention module directly to all possible features in terms of scale and task requires a high complexity, we propose to apply the attention module sequentially for the task and scale. The cross-task attention module (CTAM) is first applied to facilitate the exchange of relevant information between the multiple task features of the same scale. The cross-scale attention module (CSAM) then aggregates useful information from feature maps at different resolutions in the same task. Also, we attempt to capture long range dependencies through the self-attention module in the feature extraction network. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the NYUD-v2 and PASCAL-Context dataset. Our code is available at https://github.com/kimsunkyung/SCA-MTL
BibTeX
@inproceedings{kim2022sequential,
  author = {Sunkyung Kim and Hyesong Choi and Dongbo Min},
  title = {{Sequential Cross Attention Based Multi-Task Learning}},
  booktitle = {ICIP},
  year = {2022},
  url = {https://openreview.net/forum?id=5X0iItbXLd},
}

Loading related content...

Figure