TL;DR
The highly popular Muon optimizer fails to converge on the classic class of convex, nonsmooth Lipschitz functions—a standard theoretical benchmark used to justify optimizers like Adam and Shampoo. While adding "error feedback" mathematically fixes this convergence issue, it actually hurts Muon's performance in practice on image classification and language modeling tasks. This reveals that Muon's empirical success relies on structural properties like smoothness, making standard convex Lipschitz theory the wrong lens for understanding it.
What Problem Does This Paper Address?
Muon has recently emerged as a highly competitive optimizer for training large language models (such as DeepSeek-V4) and diffusion models. However, its theoretical foundations remain poorly understood. Most existing convergence analyses for Muon assume that the objective function is smooth (i.e., has Lipschitz continuous gradients).
In contrast, the most successful theoretical framework for designing deep learning optimizers (like AdaGrad, Adam, and Shampoo) has historically been the class of convex Lipschitz functions, which accommodates nonsmoothness. This paper investigates a fundamental question: Is the classical convex Lipschitz model a useful framework for understanding and analyzing Muon?
Methods
- Reduction to Signed Momentum: The authors prove that when optimizing functions that depend only on the diagonal entries of a matrix, Muon simplifies exactly to the coordinate-wise signed momentum method. This reduction allows them to construct a simple, tractable 2D counterexample function to analyze Muon's behavior.
- Mathematical Counterexamples: Using this 2D function, they construct two rigorous counterexamples showing that standard Muon fails to converge to the optimal solution from almost any initialization, regardless of whether the learning rate schedule is decreasing, adaptive, or tuned offline.
- Error Feedback (EF-Muon): To see if Muon can be rescued theoretically, the authors frame non-Euclidean updates (like Muon's polar factor projection) as compression operators. Borrowing techniques from distributed optimization, they introduce an error feedback mechanism that tracks the discrepancy between the true gradient and its compressed update. This yields a modified algorithm, EF-Muon, which possesses proven convergence guarantees in the convex Lipschitz setting.
Datasets
The authors evaluated the practical impact of their theoretical fix using two representative deep learning workloads:
- CIFAR-10: Used for image classification, training a WideResNet-28-10 model with 36 million parameters.
- FineWeb-Edu 10B: Used for language modeling, training a 206-million-parameter nanoGPT transformer model.
Key Insights and Ideas
- The Diagonal Bridge: Proving that Muon reduces to signed momentum on diagonal matrices provides a powerful diagnostic tool. Any future theoretical explanation of Muon's success must remain consistent with this exact specialization.
- Muon Gets Trapped: In the absence of smoothness, Muon's updates cause the iterates to get trapped in stable, non-convergent cycles or remain bounded away from the minimum indefinitely.
- The Theory-Practice Paradox: This paper highlights a fascinating mismatch between optimization theory and deep learning practice. The error feedback mechanism successfully fixes Muon's theoretical shortcomings on paper, yet completely degrades its performance on real-world neural network training.
- Smoothness is Essential: The ultimate takeaway is that Muon's real-world power does not come from handling worst-case nonsmooth landscapes. Its success is intrinsically tied to structural properties absent from the convex Lipschitz model—most plausibly, smoothness.
Key Results
- Theoretical Failure vs. Fix: The paper proves that standard Muon fails to converge on worst-case convex Lipschitz functions. As illustrated in the paper's counterexample simulation, standard Muon iterates get stuck cycling indefinitely, whereas EF-Muon successfully converges to the minimum.
- Empirical Performance Degradation:
- Image Classification (CIFAR-10): MuonMax (a regularized variant of Muon) achieved the best validation cross-entropy (CE) loss of 0.265, slightly outperforming vanilla Muon (0.287). However, the theoretically corrected EF-MuonMax performed significantly worse, trailing at 0.388.
- Language Modeling (nanoGPT): Vanilla Muon reached the best validation CE loss of 3.01, followed by MuonMax at 3.27. Once again, the error-feedback variant performed the worst, finishing at 3.43.
Limitations and Future Work
- Fixed Momentum Assumption: The paper's counterexamples rely on a fixed momentum coefficient ($\beta \in [0, 1)$), whereas some advanced convergence proofs allow or require a scheduled momentum parameter ($\beta_t \to 1$).
- Exact Polar Factorization: The analysis assumes exact calculation of the polar factor, whereas practical implementations of Muon approximate it using Newton-Schulz matrix polynomials.
- Alternative Momentum Styles: The authors only consider Polyak-style momentum, leaving open the question of whether Nesterov-style momentum alters these convergence dynamics.
- Future Direction: The authors conclude that researchers should abandon worst-case convex Lipschitz theory when analyzing Muon, and instead focus on frameworks that exploit convex smoothness or other specific structural properties of neural network loss landscapes.
Summary updated