TL;DR
Matrix-structured optimizers like Shampoo, SOAP, and Muon have shown impressive gains in LLM pretraining, but this research demonstrates that these benefits do not automatically transfer to KL distillation. In a rigorous comparison across Qwen model scales, the authors found that standard elementwise optimizers—AdamW and AdaGrad—remain the most efficient choices for short-horizon distillation, offering the best trade-off between model quality and trainer time.
What Problem Does This Paper Address?
In the world of Large Language Models (LLMs), pretraining is a marathon: models start from random initialization and train for trillions of tokens. Recently, "matrix-structured" optimizers (which treat weight layers as matrices to capture correlations between parameters) have proven more data-efficient than standard AdamW in this marathon.
However, Knowledge Distillation (KD) is a different beast. It is a "short-horizon" task where a smaller student model learns to mimic a larger teacher model, starting from an already instruction-tuned state. Because distillation involves expensive teacher inference and a different objective (matching probability distributions rather than just predicting the next token), it wasn't clear if the extra computational overhead of fancy matrix optimizers would actually pay off. This paper investigates whether the "pretraining hype" for these optimizers holds up in the distillation regime.
Methods
The authors conducted a head-to-head comparison of six optimizer recipes:
- References (Elementwise): AdamW and diagonal AdaGrad.
- Shampoo Reductions: Two tractable versions of the Shampoo optimizer designed to fit within memory constraints: a "scalar-row-mean quarter-power" version and a "right-only half-power" version.
- SOAP: An optimizer that rotates gradients into a Shampoo-style basis and then applies Adam.
- All-2D Muon: A variant of the Muon optimizer that applies spectral updates (orthogonalization) to all two-dimensional parameters, including embeddings and output heads.
The study used a forward-KL objective at temperature 1.0, training for a fixed budget of 1,683 updates (roughly three effective epochs). They evaluated performance based on Validation KL (lower is better) and Trainer Time (wall-clock time including overhead).
Datasets
The experiments utilized NVIDIA’s Nemotron-Post-Training-Dataset-v2, which consists of chain-of-thought (CoT) traces.
- Training/Validation: The data was split into ~80k training examples and ~1k unique validation examples.
- Weighting: The loss function used a "blended" approach, weighting the "reasoning" (thought process) and "answer" regions of the traces equally, regardless of their token length.
- Models: The researchers tested four dense Qwen student-teacher pairs, ranging from a 0.6B student (1.7B teacher) up to a 32B student (72B teacher).
Key Insights and Ideas
- The Transfer Gap: Gains seen in pretraining (from-scratch training) are not "portable." The short-horizon nature of distillation means there isn't enough time for matrix-structured optimizers to overcome their initial setup costs or slower per-step speed.
- Overhead is the Dealbreaker: Matrix-structured optimizers add significant "trainer-time" overhead. For example, SOAP was roughly 2 to 3 times slower than AdamW. In a resource-constrained environment, you could often train a model for more steps with AdamW and get a better result than fewer steps with a "smarter" optimizer.
- Recipe Sensitivity: The two Shampoo variants actually reversed their performance ranking depending on which model pair was being trained. This suggests that there is no "universal" matrix optimizer configuration; performance is highly dependent on specific model geometry.
- AdaGrad’s Resurgence: While AdamW is the industry standard, diagonal AdaGrad was slightly faster than AdamW at larger scales (14B and 32B) while maintaining comparable quality, likely due to its simpler state (one accumulator instead of two).
Key Results
- Quality: AdamW generally attained the lowest validation KL across all scales. At the 0.6B scale, SOAP and Muon were competitive, but they failed to pull ahead as the models grew.
- Speed: Every matrix-structured method increased trainer time relative to AdamW.
- Scaling: In the largest test (Qwen2.5 32B student), SOAP was excluded from the full-budget summary because its measured trainer-time overhead was so substantial (taking 4.31x longer than AdamW for a matched-step run).
- Trajectories: Trajectory plots showed that while some optimizers (like AdaGrad or Muon) might start strong, AdamW consistently caught up or surpassed them by the end of the short training budget.
Limitations and Future Work
- Scope: The study focused exclusively on dense Qwen models and chain-of-thought data. Results might differ for sparse architectures (like Mixture-of-Experts) or different data domains.
- Horizon: The "short-horizon" was limited to ~3 epochs. It remains possible that matrix-structured optimizers could show benefits if distillation were extended to a much longer horizon.
- Implementation: The authors note that their findings are tied to specific FSDP2 (Fully Sharded Data Parallel) implementations. Future work could explore if more optimized kernels for matrix operations (like those in JAX) might narrow the trainer-time gap.
- Downstream Tasks: The paper measures Validation KL (how well the student mimics the teacher's probabilities) but does not measure downstream task accuracy (e.g., MMLU scores), which is the ultimate goal of distillation.
Summary updated