Careful with that Scalpel: Improving Gradient Surgery with an EMA

Yu-Guan Hsieh, James Thornton, Eugene Ndiaye, Michal Klein, marco cuturi, Pierre Ablin

Abstract

Gradient surgery techniques, such as PCGrad, aim to resolve conflicts between multiple optimization objectives by projecting gradients. However, these methods often fail in stochastic settings because the non-linear projection of a noisy mini-batch gradient does not equal the projection of the true underlying gradient. We introduce Bloop (BiLevel Optimization with Orthogonal Projection), a method that stabilizes gradient surgery using an Exponential Moving Average (EMA). Bloop effectively prioritizes a primary task while optimizing auxiliary goals, achieving superior Pareto fronts across diverse vision and NLP benchmarks.

Introduction: The Conflict in Multi-Objective Optimization

In modern deep learning, models are rarely trained on a single objective. Beyond the primary task (e.g., cross-entropy), we often incorporate auxiliary losses for regularization, robustness, or specialization. The standard approach, "Mixed Training," simply sums these losses with a weighting factor $\lambda$. However, when gradients from different tasks conflict, they can interfere, leading to suboptimal convergence.

"Gradient surgery" techniques were developed to address this by projecting conflicting gradients. While theoretically sound in full-batch settings, they struggle with the stochasticity of mini-batch training. In high-noise regimes, standard surgery often collapses back into simple mixed training, losing its intended benefits.

Methods: Stabilizing Surgery with Bloop

Bloop treats multi-objective training as a bilevel optimization problem: minimize the auxiliary loss within the solution space that minimizes the main loss.

Orthogonal Projection

Bloop calculates an update direction that follows the main gradient exactly but adds a component of the auxiliary gradient that is strictly orthogonal to it. This ensures that auxiliary optimization does not impede progress on the primary task at a first-order level.

Conceptual diagram of the Bloop orthogonal projection method, showing the main gradient ($g_{main}$), auxiliary gradient ($g_{aux}$), and the resulting projected direction.

The EMA "Secret Sauce"

To overcome the noise inherent in stochastic training, Bloop maintains an Exponential Moving Average (EMA) of the main task's gradients. Instead of projecting the auxiliary gradient onto a noisy mini-batch gradient, it projects it onto this stabilized EMA. This stabilization is computationally efficient, requiring only two additional dot products per step, making it nearly as fast as standard regularized training.

Key Insights: The Pitfall of Stochasticity

A core contribution of this work is identifying why previous surgery methods fail. In high-dimensional spaces, the expected value of a standard projection collapses under noise. As mini-batch variance $\sigma$ increases, the projected direction of standard methods regresses toward the simple mixed training direction.

The impact of gradient noise ($\sigma$) on update directions. As noise increases, the expected value of standard mini-batch projections (blue) deviates from the ideal Bloop direction and converges toward the mixed training direction (orange).

By using an EMA, Bloop remains robust to this noise, maintaining a consistent projection direction even when individual mini-batches are highly variable. This explains why previous methods like PCGrad often perform no better than simple weight decay in deep learning practice.

Experimental Results

The authors evaluated Bloop across several domains, including Computer Vision (MNIST, ImageNet), Multi-task Learning (Cifar10Mnist), and NLP (Transformer pre-training on C4 and translation specialization on ParaCrawl/WMT).

Bloop consistently achieved superior Pareto fronts. For any given performance level on the main task, Bloop reached a lower auxiliary loss than Mixed Training, PCGrad, or Dynamic Barrier methods. The gains were particularly notable in Large Language Model (LLM) scenarios, where Bloop improved specialization on high-quality data without sacrificing general performance.

Conclusion and Future Work

Bloop provides a robust framework for hierarchical multi-task learning by addressing the fundamental flaw of stochastic gradient surgery. While currently designed for primary/secondary hierarchies, future work could extend this to "tri-level" optimization for complex pipelines involving pre-training, fine-tuning, and safety alignment. The key takeaway is that for gradient surgery to be effective in the real world, the "scalpel" must be stabilized against the noise of stochastic optimization.

Summary updated

Abstract

Beyond minimizing a single training loss, many deep learning estimation pipelines rely on an auxiliary objective to quantify and encourage desirable properties of the model (e.g. performance on another dataset, robustness, agreement with a prior). Although the simplest approach to incorporating an auxiliary loss is to sum it with the training loss as a regularizer, recent works have shown that one can improve performance by blending the gradients beyond a simple sum; this is known as *gradient surgery*. We cast the problem as a constrained minimization problem where the auxiliary objective is minimized among the set of minimizers of the training loss. To solve this bilevel problem, we follow a parameter update direction that combines the training loss gradient and the orthogonal projection of the auxiliary gradient to the training gradient. In a setting where gradients come from mini-batches, we explain how, using a moving average of the training loss gradients, we can carefully maintain this critical orthogonality property. We demonstrate that our method, Bloop, can lead to much better performances on NLP and vision experiments than other gradient surgery methods without EMA.
BibTeX
@inproceedings{hsieh2024careful,
  author = {Yu-Guan Hsieh and James Thornton and Eugene Ndiaye and Michal Klein and marco cuturi and Pierre Ablin},
  title = {{Careful with that Scalpel: Improving Gradient Surgery with an EMA}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=IgwtflILyj},
}
Figure