TL;DR
As Large Language Models (LLMs) grow wider, they don't necessarily use that extra space effectively. This paper uncovers an Asymmetric Spectral Scaling Law: widening Feed-Forward Networks (FFNs) primarily adds low-energy "noise" directions while the core "dominant" subspace saturates quickly, leading to significant representational waste.
What Problem Does This Paper Address?
We know that bigger models generally perform better, but we don't really know why or how they use their internal "latent space." Currently, the width of FFNs—which can account for up to 67% of an LLM's parameters—is determined by arbitrary rules of thumb (like "make it 4x the embedding dimension").
The authors argue that existing scaling laws treat models as black boxes, focusing only on the relationship between size and loss. They aim to peek inside the box to see if widening a model actually expands its usable capacity or if we are just adding "empty calories" to the architecture.
Methods
The researchers performed a "spectral audit" of the covariance matrices of FFN activations across different model families (LLaMA, GPT-2, and nGPT). They introduced a diagnostic suite of four lightweight metrics:
- Hard Rank (Participation Ratio): Measures the dimensionality of the "high-energy" dominant modes—the core directions the model relies on.
- Soft Rank (Shannon Rank): Measures how uniformly variance is spread across all available directions, including the "tail."
- Spectral Concentration: Quantifies how much variance is "front-loaded" into the top eigenvalues.
- Spectral Utilization Index (SUI): A composite metric (harmonic mean of hard and soft ranks) designed to identify the "sweet spot" of balanced utilization.
By sweeping FFN widths (from 1x to 8x the embedding dimension) and training models from scratch, they could track how these spectral properties evolve during training.
Datasets
The models (LLaMA-70M, 130M, and 250M) were trained from scratch on the C4 (Colossal Clean Crawled Corpus) dataset. Evaluation was performed using standard Perplexity (PPL) metrics to correlate spectral health with actual model performance.
Key Insights and Ideas
- The Asymmetric Scaling Law: The most striking finding is that Soft Rank scales almost perfectly linearly with width, but Hard Rank grows sublinearly and with high variance. In plain English: widening a model adds a massive "tail" of low-variance directions, but the "core" thinking space doesn't expand nearly as much.
- Tail-First Growth: LLMs populate low-energy directions first. This suggests that simply making a model wider provides diminishing returns for the "dominant" features that actually drive performance.
- Spectral Collapse vs. Dilution: The paper identifies two failure modes. Spectral Dilution happens when variance is spread too thin across too many directions. Spectral Collapse (seen in larger Post-LN models) occurs when the model funnels almost all its variance into just one or two directions, rendering the rest of the latent space useless.
- Normalization is the Gatekeeper: The placement of LayerNorm (Pre-LN vs. Post-LN) drastically changes how a model uses its space. Pre-LN (the industry standard) tends to over-inflate the "tail," while Post-LN can cause total spectral collapse in larger models unless stabilized with techniques like Weight Normalization.
Key Results
- Quantifiable Asymmetry: In a LLaMA-130M model, the Soft Rank power-law fit had an exponent ($\beta$) of 1.069 (near-linear), while the Hard Rank exponent was only 0.604 (highly sublinear).
- The "Sweet Spot": For LLaMA models, the authors found that increasing width beyond ~3,000 dimensions yielded sharply diminishing spectral returns, suggesting many current LLMs might be "too wide" for their own good.
- Performance Correlation: Spectral health directly predicts stability. In LLaMA-250M (Post-LN), a "collapsed" spectrum resulted in a massive perplexity spike (PPL > 1400), while adding Weight Normalization restored spectral health and dropped perplexity to 25.08.
Limitations and Future Work
- Scope: The study is limited to English-only, decoder-only models up to 250M parameters. Whether these laws hold for 70B+ parameter models or multimodal architectures remains to be seen.
- Causality: While spectral metrics correlate strongly with performance (perplexity), the authors note that a strict causal link—proving that spectral health causes better learning—is still unproven.
- Computational Cost: Performing eigendecomposition on massive activation matrices is computationally expensive, which might make real-time monitoring difficult for the largest frontier models.
The authors suggest that these findings could lead to "spectral-aware" pruning and dynamic width scheduling, where layers are widened or narrowed during training based on how effectively they are using their allocated latent space.
Summary updated