Imagine optimizing a function shaped like a long, narrow valley. If the valley runs perfectly parallel to one of your coordinate axes, finding the minimum is straightforward: you can simply walk straight down that axis. But what happens if you rotate the entire landscape by 45 degrees? Suddenly, the valley runs diagonally. If your optimization algorithm is only capable of moving along or scaling according to the standard coordinate axes, it must now take a highly inefficient, zigzagging path to make any progress.
This vulnerability is known as coordinate dependency, and its opposite—rotation invariance—is a highly prized property in classical optimization theory. A rotation-invariant algorithm behaves identically regardless of how the coordinate system is oriented. Yet, a wide range of modern optimization techniques are deeply sensitive to the coordinate basis, failing dramatically when landscapes are rotated, or conversely, succeeding precisely because they exploit a hidden axis-aligned structure in real-world problems.
The Geometry of Axis-Aligned Failure
The simplest form of coordinate dependency appears in algorithms that update one variable at a time. As surveyed by Stephen Wright in Coordinate Descent Algorithms, the efficiency of coordinate-wise updates severely degrades when variables exhibit high degrees of coupling. When the objective function contains narrow, tilted valleys, the search direction is constrained to axes that do not align with the steep descent paths of the landscape, resulting in extremely slow convergence.
This issue is not unique to gradient-based coordinate descent; it heavily impacts derivative-free and black-box optimization methods as well. Anne Auger et al. demonstrated in Experimental Comparisons of Derivative Free Optimization Algorithms that many prominent black-box solvers experience a dramatic collapse in efficiency when objective functions are rotated, revealing an implicit assumption of variable independence built into their design.
When an algorithm lacks rotation invariance, it fails to capture the joint structure or diagonal interactions of the search space. As noted by Ashok Cutkosky et al. in Blackbox optimization of unimodal functions, sequential optimization along individual axes struggles in high-dimensional spaces where parameters interact diagonally. This manifests visually in sampling patterns: Bahador Rashidi et al. point out in Cylindrical Thompson Sampling for High-Dimensional Bayesian Optimization that sampling strategies perturbing only a few dimensions at a time create rigid, cross-shaped exploration patterns. These patterns completely miss the diagonal paths of highly coupled variables.
Axis-Aligned Landscape (Easy) Rotated Landscape (Hard)
▲ ▲
│ │ │ │ \ \
│ │ │ │ \ \
──────┼───▼───▼───► ──────┼────▼───▼─►
│ │ │ │ \ \
│ │ │ │ \ \
(Updates match the valley) (Updates must zigzag inefficiently)
Similarly, coordinate-aligned interpolation and search mechanisms in evolutionary frameworks degrade when principal directions do not align with the coordinate system, as explored by Jinchun Song et al. in Lagrange Interpolation Learning Particle Swarm Optimization and by James Jian Qiao Yu et al. in A Social Spider Algorithm for Global Optimization. Without explicit modeling of coordinate dependencies—such as through covariance matrices or differential vectors—these methods remain blind to rotated landscapes.
Adaptive Optimizers and the Sign-Gradient Mismatch
In deep learning, the most widely used optimizers are adaptive methods like Adam and AdaGrad. These methods scale updates coordinate-by-coordinate based on historical gradients. While highly effective, this coordinate-wise adaptivity introduces a severe dependency on the choice of basis.
In Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients, Lukas Balles et al. show that Adam’s update direction is heavily determined by the sign of the gradient along each coordinate axis. This sign-based rule is inherently coordinate-dependent: it works optimally when the parameter axes align perfectly with the directions of steepest descent. If the parameter space is rotated, the sign vector no longer aligns well with the true gradient vector, causing a significant drop in optimization efficiency and generalization.
The authors of The Implicit Bias of AdaGrad on Separable Data, Qian Qian et al., prove that this lack of rotational invariance means the final solutions and generalization properties of adaptive methods are highly sensitive to how input features are oriented or scaled, unlike standard gradient descent which remains stable under orthogonal transformations.
The Paradox of Neural Network Geometry
This brings us to a profound theoretical paradox. If coordinate-wise adaptive optimizers are so vulnerable to rotation, why do they dominate the training of state-of-the-art neural networks like Transformers?
The answer is that real-world neural network loss landscapes are not arbitrary; they possess a highly specific, coordinate-aligned geometry.
Mohamad Amin Mohamadi et al. in Adam Exploits ℓ∞-geometry of Loss Landscape via Coordinate-wise Adaptivity and Shuo Xie et al. in Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity demonstrate that Adam's secret weapon is its ability to exploit the $\ell_\infty$-smoothness of the loss landscape. Modern neural network architectures are structured like a grid, where the loss varies in a way that favors independent coordinate updates over global vector updates. When the authors artificially rotated these loss landscapes, Adam’s performance completely collapsed, proving that its efficacy is tightly coupled to the model's specific parameterization.
As Tianyue H. Zhang concludes in Understanding Adam Requires Better Rotation Dependent Assumptions, this extreme sensitivity means that any theoretical framework attempting to explain Adam using traditional rotation-invariant assumptions will fail. The success of modern deep learning is fundamentally tied to a delicate alignment between the optimizer's standard basis and the coordinate-aligned geometry of the network architecture itself.
Key Foundational Work
Experimental Comparisons of Derivative Free Optimization Algorithms (2010) — contributed 2 concept elements to this direction.
Modelling the Population Distribution in Multi-objective Optimization by Generative Topographic Mapping (2006) — contributed 1 concept elements to this direction.