Neural scaling laws have served as the predictive engine of the foundation model era. By framing downstream performance as a power-law function of compute, parameters, and data, these laws allow researchers to project the capabilities of massive models using cheap, small-scale runs. However, an underlying assumption anchors this predictability: the architecture must remain a standard, dense, decoder-only Transformer.
As the field explores alternative paradigms—such as Mixture-of-Experts (MoE), State Space Models (SSMs), and non-standard structural shapes—this foundational assumption is fracturing. Recent research reveals that scaling laws are highly sensitive to architectural choices, meaning that a power-law recipe derived for one model lineage rarely transfers cleanly to another.
The Hidden Assumptions of Dense Scaling
To understand why scaling laws fail to generalize, we must look at how they are constructed. Classical scaling laws treat a model as a black box parameterized by total size. In Reconciling Kaplan and Chinchilla Scaling Laws, Tim Pearce and Jinyeop Song demonstrate that historical discrepancies in scaling laws often stem from subtle choices, such as whether embedding parameters are counted or the specific scale of the models being studied. Even when these laws prove robust under stress tests—as shown by Rylan Schaeffer et al. in Evaluating the Robustness of Chinchilla Compute-Optimal Scaling—their mathematical formulations assume a rigid, uniform relationship between parameter count and computational cost per token.
When we deviate from standard dense configurations, these clean mathematical relationships break down. For instance, Zhengyu Chen in Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies identifies "sub-scaling" regimes where performance gains decelerate sharply when models encounter novel or diverse neural configurations that deviate from initial empirical baselines.
The Sparse Disconnect: Mixture-of-Experts and Beyond
The most prominent challenge to universal scaling comes from sparse architectures like Mixture-of-Experts (MoE). In an MoE model, the total parameter count grows significantly, but the number of active parameters per token remains small. This decouples capacity from compute.
Consequently, standard scaling exponents fail. As Houyi Li et al. observe in Predictable Scale (Part II) --- Farseer: A Refined Scaling Law in LLMs, the coefficients governing performance gains shift dramatically when the underlying model mechanism changes. This sentiment is echoed across the literature: Margaret Li et al. note in (Mis)Fitting Scaling Laws: A Survey of Scaling Law Fitting Techniques in Deep Learning that the applicability of power-law constants to alternative structures like MoEs or State Space Models (SSMs) remains largely unmapped.
This architectural sensitivity impacts not just pretraining loss, but downstream tasks and specialized mechanisms:
- Generative Tasks: Rylan Schaeffer demonstrates in Pretraining Scaling Laws for Generative Evaluations of Language Models that while generative scaling curves can be stabilized using log-likelihood metrics, it remains uncertain whether these predictive constants hold true across sparse neural structures.
- Multi-Task and Multilingual Learning: Liang Chen et al. show in On the Pareto Front of Multilingual Neural Machine Translation that multi-task scaling relationships are unconfirmed for sparse or extreme parameter counts. Similarly, inference-time interventions like layer-skipping (Wenhao Zhu et al. in Multilingual Contrastive Decoding via Language-Agnostic Layers Skipping) and knowledge extraction (Gwangseon Jang et al. in Leveraging Pretrained Knowledge at Inference Time: LoRA-Gated Contrastive Decoding for Multilingual Factual Language Generation in Adapted LLMs) are well-understood in dense variants but remain open questions for sparse, expert-based computation.
- Distributed Training: Even when optimizing for communication efficiency across distributed islands of compute, the predictability of scaling laws remains bound to the underlying paradigm, as noted by Zachary Charles et al. in Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo.
Structural Shape and Hyperparameter Sensitivity
The breakdown of scaling laws isn't confined to radical shifts like MoE; it occurs even when tweaking the internal proportions of dense models. Scaling laws are frequently derived from narrow experimental setups where structural hyperparameters are held constant.
In Gemstones: A Model Suite for Multi-Faceted Scaling Laws, Sean Michael McLeish et al. systematically demonstrate that scaling law coefficients are highly sensitive to choices like width-to-depth ratios, MLP expansion factors, and vocabulary sizes. When these design choices are modified, universal scaling prescriptions derived from a single model family often fail to generalize.
This shape sensitivity is particularly acute when scaling model depth versus width. Standard parameterizations often struggle to maintain stable feature learning as depth increases. To address this, Nolan Simran Dey et al. propose CompleteP in Don't be lazy: CompleteP enables compute-efficient deep transformers, a parameterization that ensures non-lazy feature learning across both width and depth. Yet, the authors note that how these depth-wise prescriptions interact with sparse architectures or long-context mechanisms remains an open question. Similarly, Katie E Everett et al. in Scaling Exponents Across Parameterizations and Optimizers highlight that empirical validation of parameterization and alignment trends remains heavily restricted to decoder-only Transformers, leaving vision networks and MoEs unverified.
Data Regimes and Quality Dynamics
Finally, the interaction between architecture and data constraints introduces further non-linearities. When data is scarce or its quality varies, the underlying neural configuration dictates how gracefully a model scales.
In extreme low-data regimes, traditional power laws can collapse entirely, causing larger models to underperform smaller ones. Peihao Wang et al. show in Data Efficient Neural Scaling Law via Model Reusing that while transformers exhibit clear phase transitions that can be rectified via model reusing, the scaling behavior of alternative architectures like convolutional networks or state-space models in these low-data regimes remains unexplored.
Furthermore, when accounting for data quality, Anirudh Subramanyam et al. demonstrate in Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining that the specific exponents and constants of quality-aware scaling laws are tightly coupled to standard transformer architectures and may not remain stable for alternative structures like MoEs or SSMs.
Why It Matters
Scaling laws fail to generalize across architectures because they are not universal laws of nature; they are empirical descriptions of how a specific computational graph consumes data and propagates gradients. When we alter that graph—whether by sparsifying it with MoE routing, changing its sequential memory via SSMs, or altering its aspect ratio via width-to-depth shifts—we fundamentally change the dynamics of feature learning. Building a truly predictive theory of scaling requires moving past black-box compute counting and explicitly accounting for architectural topology.
Key Foundational Work
On the Pareto Front of Multilingual Neural Machine Translation (2023) — contributed 1 concept elements to this direction.
Scaling Exponents Across Parameterizations and Optimizers (2024) — contributed 1 concept elements to this direction.
Reconciling Kaplan and Chinchilla Scaling Laws (2024) — contributed 1 concept elements to this direction.