In traditional machine learning, we often optimize a single objective. But in Multi-Task Learning (MTL), we ask a single model to perform several tasks at once—like a vision model that must simultaneously predict depth, detect objects, and segment boundaries.
The standard approach is to sum the individual task losses into a single scalar: $L_{total} = \sum w_i L_i$. However, this creates a "tug-of-war" between tasks. If Task A has a loss magnitude of 1000 and Task B has a loss of 0.1, the gradient updates will be dominated by Task A. Task B becomes a rounding error, effectively ignored by the optimizer. To fix this, researchers traditionally spend weeks manually tuning the weights $w_i$, a process that is computationally expensive and fragile.
Recent research has shifted toward scale-invariant optimization, where the goal is to make the learning process independent of the raw numerical scale of task losses.
The Problem of Task Domination
When tasks have vastly different loss scales or gradient magnitudes, the model favors "easy" tasks or those with high-magnitude rewards. As noted in Equitable Multi-task Learning by Rui Zhang and Jun Yuan, this imbalance leads to poor generalization. Even in specialized fields like quantitative remote sensing, where models like SatelliteCalculator must estimate physical values like biomass and canopy height, the difference in numerical ranges can cause training instability if not carefully balanced.
The core intuition of scale invariance is simple: if you multiply a task's loss by 10, the model's update direction should not change.
Multi-Task Learning as a Bargaining Game
One of the most influential frameworks for achieving this balance treats MTL as a cooperative game. In Multi-Task Learning as a Bargaining Game, Aviv Navon and colleagues introduce Nash-MTL.
Instead of summing gradients, they use the Nash Bargaining Solution to find an update direction that is "proportionally fair" to all tasks. Imagine several players (tasks) trying to agree on a direction to move. Nash-MTL ensures that the chosen direction maximizes the product of the improvements for each task. Because this optimization depends on the relative improvement rather than absolute loss values, it is mathematically invariant to affine transformations of the losses. This eliminates the need for manual weight tuning, as the optimization remains consistent even when task gradients differ by several orders of magnitude.
From Scale Invariance to Transformation Invariance
While scale invariance handles linear scaling (like changing units from meters to kilometers), some tasks might have non-linear relationships. Surya Murthy, Kushagra Gupta, and their team extended this idea to monotonic transformation invariance in their work on DiBS-MTL.
DiBS-MTL is designed to be robust even if a task loss is squared, exponentiated, or subjected to any strictly increasing function. The key mechanism is the use of normalized gradients. By focusing strictly on the direction of the gradient rather than its magnitude, DiBS-MTL prevents any single task from dominating the trajectory simply because its loss function is "steeper" or "larger."
Automated Balancing and Efficiency
Other approaches focus on making the weights themselves dynamic and learnable. Liyang Liu and the authors of Towards Impartial Multi-task Learning proposed IMTL, which uses a closed-form geometric solution to balance gradients and loss scales automatically. This ensures all tasks progress at a comparable pace throughout the entire training process.
Similarly, the You Only Train Once (YOTO) framework by Christos Sakaridis treats loss weights as learnable parameters. By constraining the sum of these weights to unity, YOTO decouples the relative importance of tasks from the global learning rate. This allows the model to find the optimal balance in a single training run, bypassing the need for expensive grid searches.
Why It Works: The Geometric Intuition
The success of these methods lies in moving the optimization from the magnitude space to the direction space.
- Gradient Normalization: By scaling gradients to a unit norm, we acknowledge that the direction toward the minimum is more informative than the steepness of the current loss surface.
- Conflict Resolution: When tasks disagree on a direction, bargaining solutions like Nash-MTL find the "middle ground" that doesn't unfairly penalize tasks with smaller natural scales.
- Scale Preservation: Methods like those described in A Scale-Invariant Task Balancing Approach ensure that the total optimization signal remains stable, preventing the "exploding gradients" that often occur when multiple high-scale losses are summed.
By building these invariances directly into the optimizer, researchers are moving toward a future where multi-task models are as easy to train as single-task ones, regardless of how diverse the underlying objectives may be.