TL;DR
MuonRec introduces the recently proposed Muon optimizer to recommender systems, breaking the industry's default reliance on Adam/AdamW. By applying orthogonalized momentum updates to 2D weight matrices, MuonRec accelerates training convergence by an average of 32.4% while simultaneously boosting recommendation quality (NDCG@10) by 12.6%.
What Problem Does This Paper Address?
Recommender systems (RecSys) are undergoing a massive scaling revolution, with modern architectures expanding to over a billion parameters to handle web-scale interaction data. However, while model architectures have evolved rapidly, the choice of optimizer has remained stagnant: almost every modern RecSys pipeline defaults to Adam or AdamW.
The authors argue that this unexamined default is sub-optimal. In deep Transformer-based recommenders, coordinate-wise adaptive optimizers like Adam can bias weight updates toward a low-rank subspace, leading to "representation collapse." This paper addresses this gap by investigating whether matrix-structured optimization can overcome these limitations and provide more efficient, high-quality training for large-scale recommendation models.
Methods
The authors introduce MuonRec, a framework that adapts the Muon optimizer for recommendation tasks using a hybrid optimization strategy. Parameters are partitioned into two distinct groups:
- The Muon Group: Applied to all 2D weight matrices inside Transformer and sequence modeling blocks (such as attention projections and Multi-Layer Perceptron weights). Instead of updating weights coordinate-by-coordinate, Muon calculates gradient momentum and applies Newton-Schulz iteration to approximately orthogonalize it. This treats the parameters as linear operators, promoting diverse update directions and preventing the model from converging along only a few dominant eigenvectors.
- The Adam Group: Applied to 1D parameters (biases, LayerNorm), embeddings, and output layers (such as semantic ID embeddings and softmax heads), where matrix-structured orthogonalization does not naturally apply.
Datasets
The framework was evaluated on two public datasets from the Amazon Review collection:
- Industrial and Scientific: Comprising 1,987 users, 3,686 items, and 13,555 interactions.
- Office Products: Comprising 2,024 users, 3,459 items, and 13,929 interactions.
Both datasets were preprocessed using standard 5-core filtering and a chronological leave-one-out strategy. The authors tested MuonRec across traditional sequential recommenders (GRU4Rec, Caser, SASRec) and modern generative recommenders (TIGER and MiniOneRec, ranging from 0.5B to 3.0B parameters).
Key Insights and Ideas
- Implicit Spectral Regularization: Muon acts as an implicit spectral regularizer. By forcing updates to be closer to orthogonal, it encourages a more uniform singular-value spectrum. This is particularly beneficial for generative recommenders, as it reduces "index collapse" during tokenization.
- Hyperparameter Scale Invariance: A major headache in scaling large models is re-tuning hyperparameters. The authors discovered that Muon's optimal learning rate ($\eta \approx 10^{-2}$) remains completely stable across model scales from 0.5B to 3.0B parameters. This allows engineers to safely tune hyperparameters on small proxy models before deploying to billion-scale giants.
- Multi-Stage Trajectory Consistency: For generative models that undergo both Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), using Muon in both stages yields the best results. Introducing Muon only during the RL phase can disrupt optimization momentum because the base model was originally warmed up using Adam's dynamics.
Key Results
MuonRec consistently outperformed Adam/AdamW baselines in both speed and accuracy:
-
Faster Convergence: MuonRec reduced converged training steps by an average of 32.4% across all settings. For instance, the generative model TIGER converged roughly 3× faster when using Muon on the Industrial dataset.
Figure 1: Training dynamics of TIGER [27] on Industrial (left) and Office (right) datasets. The top row displays train- ing loss, while the bottom row reports the NDCG@10 curves. The Muon optimizer demonstrates significantly faster con- vergence and superior final performance compared to Adam. -
Better Recommendation Quality: Beyond speed, final ranking metrics saw substantial improvements. Traditional models achieved an average boost of 15.1% in Recall and 14.5% in NDCG. Generative models saw even larger jumps; TIGER achieved a massive +54.2% Recall and +56.0% NDCG on the Office dataset.
- Robust Scaling: On the 1.5B parameter MiniOneRec model, Muon cut training steps by over 25% while maintaining superior ranking performance over Adam.
Limitations and Future Work
The authors noted a performance drop when scaling MiniOneRec from 1.5B to 3.0B parameters under both optimizers. They attribute this to overfitting caused by the limited size of the evaluation datasets.
Future work will focus on testing MuonRec against much larger, web-scale datasets to unlock the full potential of the 3B+ parameter models. Additionally, the authors suggest exploring further hardware-level optimizations to reduce the computational overhead of the Newton-Schulz iterations during training.
Summary updated