Overcoming the Per-Task Gradient Bottleneck in Multi-Task Learning

156 concepts122 papers2 proposals

Multi-task learning (MTL) promises a highly efficient paradigm for artificial intelligence: a single, unified neural network capable of executing diverse tasks simultaneously. By sharing an underlying parameter space, tasks can theoretically pool their representations, leading to better generalization and reduced overall parameter footprints.

In practice, however, joint training often triggers a "tug-of-war" or "seesaw phenomenon" where tasks compete for capacity. Improving performance on one task frequently degrades performance on another due to conflicting optimization directions. To resolve these conflicts, a prominent subfield of machine learning has focused on advanced gradient manipulation techniques—often called "gradient surgery." Algorithms like CAGrad (Conflict-Averse Gradient Descent for Multi-task learning) by Bo Liu et al. and Nash-MTL (Multi-Task Learning as a Bargaining Game) by Aviv Navon et al. treat gradient aggregation as a sophisticated optimization or cooperative bargaining game to find a balanced update direction.

Yet, as models scale to billions of parameters and tasks scale from handfuls to thousands, these sophisticated gradient manipulation frameworks hit a fatal scalability wall.

The Hidden Cost of Gradient Surgery

To understand why advanced multi-task optimizers fail to scale, we must look at the mechanics of standard backpropagation versus gradient manipulation.

In standard single-task training, or simple joint training, the total loss is a scalar sum of individual losses. Because differentiation is a linear operator, the gradient of a sum is the sum of the gradients:

$$\nabla_\theta \mathcal{L}_{\text{total}} = \sum_{t=1}^T \nabla_\theta \mathcal{L}_t$$

Standard deep learning frameworks exploit this by accumulating gradients directly into a single shared vector of size $N$ (where $N$ is the number of model parameters) during a single backward pass. The memory overhead is $O(N)$, completely independent of the number of tasks $T$.

Gradient surgery algorithms break this efficiency. To detect and resolve conflicts—such as projecting conflicting gradients away from each other or computing a Pareto-optimal shared direction—the optimizer must inspect the individual gradient vector $\mathbf{g}_t = \nabla_\theta \mathcal{L}_t$ for every single task $t$ independently.

This introduces two massive bottlenecks:

  1. Computational Overhead: The system must execute $T$ distinct backward passes at every single training iteration. The computational cost scales linearly ($O(T)$) with the number of tasks.
  2. Memory Bloat: Instead of storing a single gradient vector of size $N$, the system must store $T$ separate high-dimensional vectors in GPU memory simultaneously to perform operations like orthogonal alignment or matrix decompositions. The memory overhead scales as $O(T \times N)$.

For a modern foundation model where $N$ is in the billions and $T$ represents dozens of downstream tasks or data domains, storing $O(T \times N)$ parameters causes immediate out-of-memory errors, forcing drastic reductions in batch sizes and crippling training throughput.

The Scalable Alternative: Rethinking Scalarization

Faced with this bottleneck, recent research has pivoted away from explicit gradient manipulation, focusing instead on methods that achieve balanced multi-task optimization while preserving the $O(1)$ memory efficiency of a single backward pass.

The simplest way to maintain scalability is scalarization—taking a weighted average of task losses $\sum w_t \mathcal{L}_t$. In Scalarization for Multi-Task and Multi-Domain Learning at Scale, Amelie Royer et al. demonstrate that simple scalarization is remarkably effective when scaling to large models, often matching or outperforming complex gradient manipulation techniques without any memory bloat. To find the optimal task weights $w_t$ without computing per-task gradients, the authors leverage Population-Based Training (PBT), treating weight selection as an outer-loop hyperparameter optimization problem.

Similarly, Antonio Demontis et al. in In Defense of the Unitary Scalarization for Deep Multi-Task Learning show that when paired with standard regularization techniques, basic unitary scalarization (equal weighting) remains highly robust and avoids the linear scaling costs that render specialized multi-task optimizers impractical for large-scale systems.

Gradient-Free Task Balancing

If scalarization is the path forward for scalability, how can we dynamically adjust the weights $w_t$ during training to prevent easy tasks from dominating hard ones without calculating per-task gradients? The community has developed two primary workarounds: tracking loss histories and tracking scalar gradient statistics.

Approximating Dynamics via Loss Histories

Instead of looking at the high-dimensional gradient vectors themselves, algorithms can look at the 1D scalar trajectories of the losses over time.

Bo Liu et al. introduced FAMO (FAMO: Fast Adaptive Multitask Optimization), an optimizer that dynamically adjusts task weights to ensure equal progress across tasks. Crucially, FAMO uses a mathematical shortcut that approximates the task gradient relationships using only the history of the task losses. This bypasses the need to compute or store separate gradient vectors, reducing the multi-task optimization overhead to negligible levels.

A similar philosophy underpins CoBa (CoBa: Convergence Balancer for Multitask Finetuning of Large Language Models), proposed by Hang Yu et al. Designed explicitly for large language models, CoBa monitors the slopes of validation loss curves. By dynamically adjusting task weights based on these scalar trajectories, it ensures all tasks converge at a similar pace and prevents individual tasks from overfitting, entirely avoiding the extreme memory and processing requirements of gradient-based balancing.

Leveraging Scalar Summaries

Another approach is to compute lightweight scalar summaries of the gradients rather than storing full vectors. In PiKE: Adaptive Data Mixing for Multi-Task Learning Under Low Gradient Conflicts, Zeman Li observes that gradients in large-scale models are rarely in severe conflict. By utilizing scalar summaries of gradient statistics—such as norm and variance—the algorithm optimizes data-mixing weights efficiently for models with billions of parameters without full gradient tracking.

Task Sampling and Sequential Updates

A final paradigm for scaling multi-task optimization involves modifying when and how tasks are updated, moving away from the requirement that every task must contribute to every gradient step.

Multi-Armed Bandits for Task Selection

Instead of performing a backward pass for all $T$ tasks at every iteration, systems can treat task selection as a resource allocation problem. Chenguang Wang et al. developed a training paradigm for multi-task combinatorial optimization solvers (Efficient Training of Multi-task Neural Solver for Combinatorial Optimization) that uses Multi-Armed Bandits (MAB). Guided by a theoretically derived "influence matrix," the bandit dynamically selects the single most impactful task to train at each step, eliminating the need to compute losses and gradients for the entire task set simultaneously.

Randomized Block and Group Updates

In large-scale bilevel optimization problems, such as multi-task deep AUC maximization, Quanqi Hu et al. propose single-loop stochastic algorithms (Multi-block Min-max Bilevel Optimization with Applications in Multi-task Deep AUC Maximization) that utilize randomized block sampling. By updating only a few randomly sampled tasks at each iteration, they achieve strong convergence guarantees while drastically reducing the per-iteration computational and memory footprint.

Similarly, Wooseong Jeong et al. propose sequential group updates (Selective Task Group Updates for Multi-Task Optimization). Instead of manipulating gradients across all tasks simultaneously, tasks are clustered into groups and updated sequentially. This provides a highly scalable alternative that mitigates negative transfer while maintaining the memory efficiency required for large-scale datasets.

Summary

The evolution of multi-task optimization highlights a classic deep learning trade-off between optimization precision and computational scalability. While early work focused on exact geometric solutions to gradient conflicts, the modern era of massive foundation models demands a shift toward lightweight, scalable alternatives. By substituting high-dimensional gradient tracking with scalar loss histories, population-based scalarization, and strategic task sampling, modern architectures can successfully navigate task interference without collapsing under the weight of per-task gradient overhead.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_task_optimization_scalability_d2ba1928,
  title = {multi-task optimization scalability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-task-optimization-scalability/rdr_d2ba19281523439ea22aae9de026045b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_task_optimization_scalability_d2ba1928_v1,
  title = {multi-task optimization scalability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-task-optimization-scalability/rdr_d2ba19281523439ea22aae9de026045b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake