Cross-Task Attention for Multi-Task Learning

This direction is merged into another direction: cross-task attention. Its last version stays readable and citable here.

Version 1 of 174 papers85 conceptsfrozen 2026-10-04

In traditional multi-task learning (MTL), models typically share a "backbone" of early layers and then branch off into task-specific heads. This approach, known as hard parameter sharing, assumes that tasks benefit from a shared feature representation. However, it often fails to capture the nuanced, bidirectional dependencies between tasks—for instance, how knowing the depth of a scene can help refine the boundaries of an object in semantic segmentation.

A growing body of research has moved beyond this rigid structure, proposing Cross-Task Attention. Instead of just sharing weights, these models allow tasks to "talk" to one another, using attention mechanisms to dynamically query and incorporate relevant information from parallel task branches.

The Intuition: Tasks as Mutual Scaffolds

The core insight behind cross-task attention is that related tasks provide structural scaffolding for one another. In medical imaging, Xiaomeng Li and colleagues demonstrated this with CANet, where features from one disease diagnostic task generate attention weights to modulate another. This "cross-pollination" ensures that the model's focus on a primary condition is informed by the clinical dependencies of a secondary one.

This interaction is often implemented through bidirectional cross-attention. In the Co-Interactive Transformer, Libo Qin and others show that intent detection and slot filling in natural language understanding can mutually refine each other. By using the representations of Task A as queries to attend to the keys and values of Task B, the model enforces mutual constraints, ensuring the predicted intent and the identified slots are semantically consistent.

Mechanism: Task Tokens and Global Interaction

A more recent evolution involves treating tasks as specialized "tokens" within a shared Transformer sequence. Rather than having separate branches that occasionally exchange data, these models represent each task as a learnable query or prompt.

  • Task-Specific Proxies: Xiangtai Li and Jing Zhang introduced the MQTransformer, which uses task-specific proxy queries. By concatenating these queries and applying a shared attention mechanism, the model allows tasks to exchange information at a fraction of the computational cost of pixel-level interactions.
  • Hierarchical Task Tokens: In HiTTs, Hanrong Ye and Dan Xu use compact tokens to discover missing supervision signals. These tokens model cross-task correlations, such as the alignment between depth discontinuities and semantic boundaries, even when labels are only partially available.
  • Prompt-Based Learning: Roei Herzig and Leonid Karlinsky developed PromptonomyViT, which adds auxiliary "task prompts" to video transformers. These prompts learn 3D structures (like pose or depth) from synthetic data, and through self-attention, they "inject" this geometric common sense into the primary action recognition task.

Solving the Negative Transfer Problem

A major hurdle in MTL is "negative transfer," where learning one task degrades the performance of another. Cross-task attention provides a mechanism to mitigate this by acting as a selective router.

In Divide and Merge, Omer Sahin Tas and colleagues use interactive semantic decoding to allow object detection and road mapping to share information while isolating them from unrelated motion dynamics. This selective merging ensures that "positive transfer" occurs between similar semantic domains without interference from tasks with different underlying logic.

Similarly, MulT by Tong Zhang and Sabine Süsstrunk uses a shared attention mechanism in the decoder. By pulling queries and keys from a shared encoder while keeping values task-specific, the model effectively captures synergies while maintaining the unique requirements of each output.

Why It Works: Global Context and Consistency

The success of these methods stems from their ability to model global dependencies. Traditional CNN-based MTL is often limited by local receptive fields. Transformer-based architectures like InvPT, proposed by Hanrong Ye and Dan Xu, concatenate features from multiple tasks into a single sequence. This allows the model to learn complementary relationships across the entire scene, leading to massive gains in mIoU for dense prediction.

By treating tasks as interactive entities rather than isolated outputs, cross-task attention transforms multi-task learning from a competition for shared parameters into a collaborative process of mutual refinement.

Papers in this version

Showing 60 of 74 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_task_attention_for_multi_task_lear_fe8064a5,
  title = {Cross-Task Attention for Multi-Task Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-task-attention-for-multi-task-learning/rdr_fe8064a5eaa84ee5b9a32bc984ac5200},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_task_attention_for_multi_task_lear_fe8064a5_v1,
  title = {Cross-Task Attention for Multi-Task Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-task-attention-for-multi-task-learning/rdr_fe8064a5eaa84ee5b9a32bc984ac5200/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}