Gradient Surgery for Multi-Task Learning

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn

Introduction

Multi-task learning (MTL) aims to train a single model to perform multiple tasks simultaneously, leveraging shared features to improve generalization and data efficiency. However, in practice, MTL often underperforms compared to training separate models for each task. This performance gap is frequently attributed to gradient interference, where gradients from different tasks conflict during the optimization process. This paper introduces PCGrad (Projecting Conflicting Gradients), a "gradient surgery" technique designed to mitigate these conflicts by projecting gradients onto each other's normal planes.

The "Tragic Triad" of MTL Optimization

The authors identify three specific conditions that create a destructive optimization environment in multi-task settings, termed the "Tragic Triad":

  1. Conflicting Gradients: Task gradients point in opposite directions (negative cosine similarity).
  2. High Curvature: The loss landscape contains sharp, narrow valleys.
  3. Magnitude Imbalance: One task's gradient is significantly larger than others, allowing it to dominate the update and "bully" smaller tasks.

When these conditions coincide, standard optimizers like Adam or SGD overestimate progress on the dominant task while inadvertently damaging performance on others, leading to stalled learning.

PCGrad: Gradient Surgery

The proposed solution, PCGrad, is a model-agnostic algorithm that modifies gradients before they are passed to the optimizer. The core intuition is to eliminate the component of a task's gradient that directly opposes another task's gradient.

The PCGrad Algorithm: 1. Compute individual gradients $g_i$ and $g_j$ for tasks $i$ and $j$. 2. Determine if a conflict exists by checking if their dot product is negative ($g_i \cdot g_j < 0$). 3. If they conflict, project $g_i$ onto the normal plane of $g_j$: $$g_i = g_i - \frac{g_i \cdot g_j}{\|g_j\|^2} g_j$$ 4. This process is repeated for all task pairs in a randomized order to ensure an unbiased update.

The PCGrad surgery: projecting conflicting gradient $g_i$ onto the normal plane of $g_j$ to create a non-conflicting update.

Optimization Landscape Analysis

To visualize the impact of PCGrad, the authors analyzed a toy multi-task objective. In standard optimization, the path often gets stuck or oscillates in high-curvature regions where task gradients conflict. PCGrad allows the optimizer to navigate these "valleys" more effectively by ensuring that the update direction does not actively harm any individual task's objective.

Visualization of the multi-task objective landscape (a-c) and optimization paths for Adam (d) vs. Adam + PCGrad (e). PCGrad successfully navigates the conflicting region to reach the global optimum.

Experimental Results

The effectiveness of PCGrad was demonstrated across several domains:

Reinforcement Learning

In the Meta-World MT50 benchmark, which involves 50 distinct robotic manipulation tasks, a standard Soft Actor-Critic (SAC) agent solved fewer than 40% of tasks. Integrating PCGrad increased the success rate to ~70%, outperforming even independent agents trained on single tasks. This suggests that PCGrad is crucial in RL, where gradient noise from complex tasks can otherwise prevent the learning of simpler ones.

Supervised Learning

  • NYUv2 & CityScapes: When combined with state-of-the-art architectures like MTAN, PCGrad achieved top results in multi-task scene understanding (segmentation, depth, and surface normals).
  • CIFAR-100: PCGrad improved test accuracy from 67.7% to 77.5% when used with routing networks, acting as a powerful regularizer that forces the model to find universally useful representations.

Conclusion

PCGrad offers a simple yet effective way to handle the inherent tensions in multi-task optimization. By performing "surgery" on conflicting gradients, it prevents destructive interference and enables models to find better minima. While it introduces some computational overhead due to per-task gradient computation, the gains in generalization and stability—particularly in complex RL environments—make it a vital tool for modern MTL research.

Summary updated

Abstract

While deep learning and deep reinforcement learning (RL) systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge. Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efficient learning. However, the multi-task setting presents a number of optimization challenges, making it difficult to realize large efficiency gains compared to learning tasks independently. The reasons why multi-task learning is so challenging compared to single-task learning are not fully understood. In this work, we identify a set of three conditions of the multi-task optimization landscape that cause detrimental gradient interference, and develop a simple yet general approach for avoiding such interference between task gradients. We propose a form of gradient surgery that projects a task's gradient onto the normal plane of the gradient of any other task that has a gradient. On a series of challenging multi-task supervised and multi-task RL problems, this approach leads to substantial gains in efficiency and performance. Further, it is model-agnostic and can be combined with previously-proposed multi-task architectures for enhanced performance.
BibTeX
@inproceedings{yu2020gradient,
  author = {Tianhe Yu and Saurabh Kumar and Abhishek Gupta and Sergey Levine and Karol Hausman and Chelsea Finn},
  title = {{Gradient Surgery for Multi-Task Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=020HornOxJ0},
}
Figure