Muon with Spectral Guidance: Efficient Optimization for Scientific Machine Learning

Binghang Lu, Jiahao Zhang, Guang Lin

TL;DR

This paper introduces SpecMuon, a novel optimizer for scientific machine learning that combines spectral orthogonalization with an energy-based relaxation mechanism. By adapting updates to the singular value spectrum of gradients and regulating step sizes based on energy, SpecMuon achieves faster convergence and improved stability for physics-informed neural networks and neural operators, especially in challenging, ill-conditioned scenarios.

What Problem Does This Paper Address?

Training physics-informed neural networks (PINNs) and neural operators often faces significant optimization hurdles. These include ill-conditioned gradients, multi-scale spectral behavior, and stiffness introduced by physical constraints. Standard optimizers like Adam struggle with these issues, leading to slow convergence, instability, and sensitivity to hyperparameters. While the "Muon" optimizer offers geometric conditioning by operating in the singular-vector basis of gradients, its aggressive updates and lack of explicit stability guarantees can be problematic. This paper aims to bridge the gap between geometric conditioning and energy stability for more robust and efficient optimization in these demanding applications.

Methods

The core innovation is SpecMuon, an optimizer that builds upon the Muon optimizer's spectral orthogonalization. Muon decomposes the gradient matrix into its singular vectors and updates parameters along these directions, effectively reshaping the optimization geometry. However, it discards singular value magnitudes, which can lead to overly aggressive steps.

SpecMuon addresses this by integrating a mode-wise relaxed scalar auxiliary variable (RSAV) mechanism. This means:

  • Spectral Decomposition: The gradient matrix is decomposed using Singular Value Decomposition (SVD) into its dominant singular directions (modes).
  • Mode-wise RSAV: For each singular direction, a scalar auxiliary variable is introduced. This variable is coupled to the loss energy and used to adaptively regulate the step size for that specific mode.
  • Relaxation: A "relaxed" update ensures that the auxiliary variable remains tied to the current loss level, providing stability.
  • Unified Approach: This approach interprets optimization as a multi-mode gradient flow, allowing principled control over stiff spectral components while preserving the geometric benefits of Muon.

Theoretically, the paper establishes modified energy dissipation laws, positivity and boundedness of auxiliary variables, and global convergence with a linear rate under the Polyak–Łojasiewicz condition.

Datasets

The paper evaluates SpecMuon on several benchmark problems commonly used in scientific machine learning:

  • Linear Regression: A controlled convex setting to examine spectral effects.
  • One-dimensional Viscous Burgers' Equation: A nonlinear PDE solved using Physics-Informed Neural Networks (PINNs).
  • Burgers' Equation (DeepONet): An operator learning task using DeepONets.
  • Two-dimensional Heat Equation (fPINN-DeepONet): A fractional PDE solved with a combination of PINNs and DeepONets.

No new datasets were introduced; standard benchmarks were used to demonstrate the optimizer's effectiveness.

Key Insights and Ideas

  • Spectral Geometry Matters: Even in simple convex problems like linear regression, the spectral properties of the gradient significantly impact optimization efficiency.
  • Energy-Based Control for Stability: Directly linking optimization steps to the loss energy (via auxiliary variables) provides crucial stability guarantees, especially for stiff problems.
  • Mode-wise Adaptation is Key: Treating different spectral components of the gradient independently allows for fine-grained control, damping stiff modes without hindering progress on well-conditioned ones.
  • Unifying Geometry and Energy: SpecMuon's core contribution is the successful integration of spectral orthogonalization (geometry) with energy-based regulation (stability), a combination previously lacking.
  • Theoretical Guarantees: The rigorous analysis provides confidence in SpecMuon's convergence properties, including linear convergence under common assumptions.

Key Results

SpecMuon consistently outperformed Adam, AdamW, and the original Muon optimizer across various benchmarks:

  • Faster Convergence: Achieved significantly faster convergence in training loss trajectories for PINNs and DeepONets.
  • Improved Stability: Maintained lower loss and exhibited more stable convergence behavior, especially in the transient regime.
  • Reduced Error: For the Burgers' equation PINN, SpecMuon produced a more accurate and smoother approximation of the reference solution, with a lower L1 error distribution.
  • Quantitative Improvements: In the Burgers' PINN experiment, SpecMuon achieved the lowest final training loss (1.55 ± 0.04 × 10⁻⁴) and the smallest mean squared error (5.06 × 10⁻⁴) compared to other methods. Similar improvements were observed for DeepONet and fPINN-DeepONet experiments.
  • Computational Cost: While SpecMuon incurs a moderate overhead due to spectral decomposition and auxiliary variable updates, this is often justified by the improved accuracy and stability.

Limitations and Future Work

  • Limitations:
    • The current analysis focuses on the core Muon-RSAV update without momentum, gradient normalization, or heuristic stabilizers. The full Algorithm 1 includes these for practical performance.
  • The paper notes that the factor $c_0 = \cos^2\angle(r_i^{k+1}, \sigma_i^k)$ measures correlation between predictors and gradient spectrum. While empirically positive, its behavior in extreme cases might warrant further study.
  • Future Work:
    • Stochastic/Mini-batch Settings: Extending SpecMuon to stochastic or mini-batch settings and exploring its interaction with variance reduction techniques for large-scale applications.
  • Adaptive Spectral Mode Selection: Developing strategies to adaptively select the number of retained spectral modes to further improve efficiency and robustness.
  • Sharper Theoretical Analysis: Investigating sharper convergence rates under weaker assumptions and a deeper analysis of spectral evolution across iterations.
  • Broader Applications: Applying the framework to more complex operator learning problems, inverse problems, and multi-physics systems to uncover additional advantages.

Summary updated

BibTeX
@article{lu2026muon,
  author = {Binghang Lu and Jiahao Zhang and Guang Lin},
  title = {{Muon with Spectral Guidance: Efficient Optimization for Scientific Machine Learning}},
  journal = {Journal of Computational Physics},
  year = {2026},
  doi = {10.1016/j.jcp.2026.115231},
}

Loading related content...

Figure