Multi-task learning (MTL) operates on an elegant premise: by training a single model on multiple related objectives simultaneously, the model can leverage shared representations to generalize better on each individual task. However, this premise relies on a fragile assumption—that the tasks are fundamentally cooperative.
When tasks are unrelated, weakly correlated, or contradictory, forcing them to share a parameter space can backfire catastrophically. Instead of synergy, the model experiences negative transfer (or task interference), where performance on individual tasks degrades compared to training them independently.
Why Negative Transfer Occurs
To understand why negative transfer happens, consider how a neural network updates its weights. In a standard hard-parameter sharing architecture, the gradients from all tasks are aggregated to update a shared backbone. If Task A requires the network to focus on fine-grained textual syntax while Task B requires a high-level semantic summary, their optimization trajectories may pull the shared weights in opposite directions.
As Liqiang Xiao et al. note in Learning What to Share: Leaky Multi-Task Network for Text Classification, a signal that is highly informative for one specific goal can introduce destructive noise into a secondary task. Standard architectures often lack the capacity to autonomously reject this unhelpful information, leaving them vulnerable to cross-task contamination, a vulnerability highlighted by Rui Mao et al. in Bridging Towers of Multi-task Learning with a Gating Mechanism for Aspect-based Sentiment Analysis and Sequential Metaphor Identification.
Discovering Latent Task Structure
One of the most intuitive ways to prevent negative transfer is to break the assumption of universal relatedness. Instead of forcing all tasks into a single shared bucket, models can automatically discover the underlying relationships between tasks and group them accordingly.
For instance, Edwin V. Bonilla et al. introduced a Gaussian Process framework in Multi-task Gaussian Process Prediction that uses a free-form covariance matrix to learn inter-task dependencies directly from data, ensuring information is only shared when tasks are genuinely correlated. In a similar vein, Christian Widmer et al. framed structure discovery as a Multiple Kernel Learning problem in Inferring latent task structure for Multitask Learning by Multiple Kernel Learning, allowing the model to uncover latent task hierarchies from scratch.
To make this grouping flexible, Abhishek Kumar proposed the GO-MTL framework in Learning Task Grouping and Overlap in Multi-task Learning, which models task parameters as sparse combinations of latent basis tasks. This allows for a "continuum" of sharing where tasks can belong to overlapping groups, preventing unrelated noise from bleeding across distinct domains.
Crucially, the authors of Which Tasks Should Be Learned Together in Multi-task Learning? demonstrated that task relationships observed in sequential transfer learning are poor predictors of simultaneous multi-task compatibility, reinforcing the need for specialized joint-training grouping mechanisms.
Gating and Selective Feature Routing
Rather than segregating tasks into rigid groups, another approach is to allow tasks to share a common network architecture but introduce dynamic "valves" that control the flow of information. This is the intuition behind gating mechanisms.
In Learning What to Share: Leaky Multi-Task Network for Text Classification, Liqiang Xiao et al. designed a "Leaky Unit" inspired by Gated Recurrent Units (GRUs) to selectively filter feature flows between task-specific networks. Similarly, Rui Mao et al. developed the Gated Bridging Mechanism in Bridging Towers of Multi-task Learning with a Gating Mechanism for Aspect-based Sentiment Analysis and Sequential Metaphor Identification to filter out noise from auxiliary tasks before it can degrade the primary objective tower. By giving the network the capacity to actively reject unhelpful features, gating mechanisms transform multi-task sharing from a passive vulnerability into an active, selective process.
Dynamic Loss Balancing
Task interference doesn't just happen because features are incompatible; it also occurs when one task dominates the optimization landscape. If Task A has a much larger loss or gradient magnitude than Task B, the shared parameters will be updated almost exclusively in favor of Task A, causing Task B's performance to plummet.
To counteract this gradient imbalance, Shengchao Liu et al. proposed Loss-Balanced Task Weighting (LBTW) in Loss-Balanced Task Weighting to Reduce Negative Transfer in Multi-Task Learning. LBTW dynamically adjusts the weight of each task's loss during training based on its rate of improvement. If a task is learning slowly or struggling due to interference, its weight is automatically increased, ensuring that no single objective completely hijacks the shared representations.
Key Foundational Work
Multi-task Gaussian Process Prediction (2007) — contributed 2 concept elements to this direction.