The Coordinate Bias of Adaptive Optimization

This direction is split into other directions: coordinate dependency. Its last version stays readable and citable here.

Version 1 of 141 papers72 conceptsfrozen 2026-10-04

In the standard theory of optimization, we often treat the coordinate system of a problem as an arbitrary choice. We assume that if an algorithm works well on a function $f(x)$, it should work just as well on $f(Rx)$, where $R$ is a rotation matrix. This property, known as rotation invariance, is a hallmark of Stochastic Gradient Descent (SGD).

However, the most successful optimizers in modern deep learning—Adam, AdaGrad, and their variants—are fundamentally not rotation-invariant. Recent research suggests that their success is not despite this "flaw," but because of it. These algorithms exploit a specific, grid-like geometry in neural network loss landscapes that SGD ignores.

The Geometry of the Grid

Standard adaptive optimizers perform coordinate-wise adaptivity. They maintain a separate learning rate for every single parameter, scaling updates based on the history of gradients in that specific dimension.

Mohamad Amin Mohamadi, Shuo Xie, and Zhiyuan Li argue in Adam Exploits ℓ∞-geometry of Loss Landscape via Coordinate-wise Adaptivity that this mechanism allows Adam to exploit the $\ell_\infty$-geometry of the loss landscape. In many real-world problems, the "smoothness" of the loss function is not uniform in all directions. Instead, the landscape is often smoother along the individual coordinate axes than it is along arbitrary diagonal directions.

By scaling updates independently for each axis, Adam can take larger steps in directions where the gradient changes slowly and smaller steps where it changes rapidly. This "diagonal preconditioning" is highly effective, but it relies entirely on the assumption that the "important" directions of the landscape are aligned with the parameters we've chosen to optimize.

The Rotation Collapse

If the effectiveness of an optimizer depends on the coordinate system, what happens if we change that system?

Researchers like Alan Milligan, Lucas Maes, and Damien Scieur have explored this through basis sensitivity analysis. In Understanding Adam Requires Better Rotation Dependent Assumptions, they demonstrate that if you apply a random rotation to the parameter space of a Transformer, Adam’s performance collapses. While SGD remains unaffected by the rotation, Adam loses its ability to converge efficiently.

This sensitivity reveals a "secret sauce" in neural network architecture: models like Transformers and CNNs appear to have a natural alignment between their architectural components (like weights in a specific layer) and the coordinate axes. When we rotate the parameters, we break this alignment, and the coordinate-wise logic of Adam becomes a hindrance rather than a help. As Lukas Balles and Philipp Hennig note in Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients, sign-based updates (which Adam uses) implicitly assume the Hessian of the loss function is axis-aligned. When the principal directions of curvature are rotated, these methods produce inefficient, oscillating trajectories.

Implicit Bias and Generalization

The choice of coordinate system doesn't just affect how fast a model trains; it affects what the model learns.

Qian Qian and Xiaoyuan Qian show in The Implicit Bias of AdaGrad on Separable Data that the lack of rotational invariance in adaptive methods leads to a different implicit bias than SGD. Because their scaling factors are tied to feature axes, the final solution direction depends heavily on the initial coordinate system.

This can be a double-edged sword. In How do simple rotations affect the implicit bias of Adam?, Adela Frances DePavia and Rebecca Willett find that Adam typically favors "rich," complex decision boundaries that generalize well. However, even a tiny rotation of the input data can cause this bias to collapse, forcing Adam to settle for a poor linear classifier. This suggests that Adam’s superior generalization is not an intrinsic property of the algorithm, but a result of how it interacts with the specific orientation of common datasets.

Beyond Deep Learning

This "axis-alignment bias" is not unique to gradient-based optimization; it is a recurring theme across the broader field of Black-Box Optimization (BBO).

  • Evolutionary Algorithms: Many popular solvers like Particle Swarm Optimization (PSO) or Differential Evolution (DE) struggle when variables are highly correlated. Zhi-Zhong Liu and colleagues point out in An Adaptive Framework to Tune the Coordinate Systems in Evolutionary Algorithms that these algorithms often search along fixed axes. When the optimal path is diagonal, they move inefficiently.
  • Bayesian Optimization: In high-dimensional spaces, Bayesian Optimization often assumes that only a few "active" dimensions matter. However, Frank Hutter and Ziyu Wang 0001 show in Bayesian Optimization in a Billion Dimensions via Random Embeddings that if these important factors are linear combinations of variables (rotated) rather than single axes, standard methods fail. Their solution, REMBO, uses random embeddings to remain invariant to these rotations.

Restoring Invariance

Recognizing the fragility of coordinate-wise adaptivity has led to new "rotation-aware" algorithms.

For geometric data, where orientation is physically meaningful, Selena Ling introduced VectorAdam for Rotation Equivariant Geometry Optimization. Instead of scaling each scalar coordinate (x, y, z) independently—which creates axis-aligned artifacts—VectorAdam performs updates on the entire vector, ensuring the optimization respects the underlying geometry.

Similarly, Adela Frances DePavia proposes EGOP-reparameterization in Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization to restore rotation invariance to adaptive methods, protecting them from performance collapse when the data orientation is unknown or arbitrary.

Ultimately, the research suggests a shift in how we view optimizers. We can no longer assume that "better" algorithms are those that ignore the coordinate system. Instead, the goal is to understand when the grid helps us—and when we need to break free from it.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_coordinate_bias_of_adaptive_optimiza_acba49d5,
  title = {The Coordinate Bias of Adaptive Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-coordinate-bias-of-adaptive-optimization/rdr_acba49d5073b443685dc8d8d7ad16457},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_coordinate_bias_of_adaptive_optimiza_acba49d5_v1,
  title = {The Coordinate Bias of Adaptive Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-coordinate-bias-of-adaptive-optimization/rdr_acba49d5073b443685dc8d8d7ad16457/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}