Relative Scaling Laws for LLMs

William Barr Held, David Leo Wright Hall, Percy Liang, Diyi YangSubmission to ICLR 2026 · Desk rejected

TL;DR

Aggregate scaling laws hide whether performance gaps between subgroups shrink or widen as compute grows. By introducing relative scaling laws—fitting power laws to the error ratios between conditions—this paper demonstrates across 255 compute-controlled models that scale naturally closes gaps in academic domains (MMLU), but exacerbates disparities across global English dialects and selectively amplifies capability/influence AI risks over adversarial ones.

What Problem Does This Paper Address?

Standard neural scaling laws typically show smooth power-law reductions in average loss as models scale up in compute, parameters, and tokens. However, aggregate benchmarks mask disparities across heterogeneous subpopulations:

  1. Does scaling act as a "universal equalizer" that lifts neglected domains to parity, or does it widen the divide?
  2. Performance differences at small scales are often confounded by inherent data entropy rather than responsiveness to scale.

Pointwise evaluations at a single scale cannot distinguish whether an observed gap is permanent or destined to vanish under the "bitter lesson" of additional compute.

Methods

The authors formalize relative scaling laws by modeling the ratio of test errors between a treatment distribution and a baseline: $$G(F) = \frac{E_{\text{treatment}}(F)}{E_{\text{baseline}}(F)} = \gamma F^{\Delta\beta}$$ - $\gamma = \alpha_{\text{treatment}}/\alpha_{\text{baseline}}$ captures the initial small-scale disparity. - $\Delta\beta = \beta_{\text{baseline}} - \beta_{\text{treatment}}$ is the relative scaling exponent. - $\Delta\beta < 0$: The treatment improves faster; the performance gap narrows (convergence). - $\Delta\beta > 0$: The treatment improves slower; the gap widens (divergence). - $\Delta\beta = 0$: The gap remains constant.

To ensure rigorous scaling comparisons, the authors train 255 decoder-only Transformers (Qwen 3 architecture) across matched-compute (IsoFLOP) budgets ranging from $10^{18}$ to $10^{20}$ FLOPs using hyperparameter schedules governed by $\mu\text{P}$ principles and AdamW augmented with AdamC and Caution.

To solve the issue of erratic, uncalibrated downstream metrics, the authors introduce a modified MCQ format that evaluates log-probabilities over the full option text (e.g., "A. Nitrogen") rather than bare option letters. This eliminates surface form competition and yields smooth compute-to-loss scaling ($R^2 = 0.61$) that accurately predicts downstream accuracy through a sigmoidal calibration curve.

       Modified MCQ Format (Smooth Loss & Predictable Scaling)
  ┌──────────────────────────────────────────────────────────────┐
  │ Question: Which gas is most abundant in Earth's atmosphere?  │
  │ Prompt Options: A. Nitrogen  B. Oxygen  C. Argon             │
  │ Scored Target Tokens: {"A. Nitrogen", "B. Oxygen", ...}     │
  └──────────────────────────────┬───────────────────────────────┘
                                 │
                                 ▼
   [ Compute (FLOPs) ] ──(Power Law)──► [ Task Loss ] ──(Sigmoid)──► [ Downstream Accuracy ]

Datasets

  • Pretraining Datasets: 85 models were trained on each of three corpora representing different data curation philosophies:
  • CommonPile: Openly licensed and public domain text.
  • DCLM Baseline: Filtered and deduplicated web-crawl data.
  • Nemotron-CC: Hybrid web text and synthetic rephrasings.
  • Evaluation Benchmarks:
  • MMLU: Academic knowledge split across STEM, Humanities, Social Sciences, and Miscellaneous subjects.
  • International Corpus of English (ICE): Written dialects across 10 regions (USA, Canada, Nigeria, Singapore, Sri Lanka, Hong Kong, India, Ireland, Jamaica, Philippines, Trinidad & Tobago).
  • Anthropic Advanced AI Risk Evaluations: 154 model-written risk datasets clustered into five high-level categories: Self-Improvement (baseline), Influence, Self-Replication, Scheming, and Incorrigibility.

Key Insights and Ideas

  • Scale narrows academic knowledge disparities: Regardless of training data bias, gaps between STEM and other domains (Humanities, Social Sciences) shrink toward parity with increased compute.
  • Scale widens dialect disparities based on population: In written English dialects, scaling does not uniformly close gaps. Dialect improvements strongly correlate with the size of each region's online English-speaking population ($R = 0.82\text{--}0.84$). Canada and Singapore track or gain on U.S. English, while Nigeria and Sri Lanka fall further behind.
  • Parameter scaling drives dialect divergence, not token scaling: Isolating model parameters vs. training tokens reveals that scaling parameters shifts relative dialect trajectories, whereas scaling data tokens simply shifts all dialects in parallel.
  • AI risks split during pretraining: As compute scales, models naturally assign higher probability to behaviors associated with capability and expansion (Self-Improvement, Influence, Self-Replication). In contrast, adversarial behaviors (Scheming, Incorrigibility) stay flat or decline during pretraining.

Key Results

  • MMLU Convergence: In CommonPile, Humanities loss starts 29% worse than STEM at $10^{18}$ FLOPs but converges to within 5% at $10^{20}$ FLOPs. On DCLM and Nemotron, Social Sciences and Miscellaneous reach near-parity with STEM.
  • Global English Divergence: Every 10× increase in a country’s online speaker population yields a 0.3% to 0.4% improvement in its relative scaling slope against U.S. English ($p < 0.005$). In CommonPile, Singapore begins below Nigeria at $10^{18}$ FLOPs but overtakes it at $6 \times 10^{19}$ FLOPs.
  • AI Risk Emergence: Scheming and Incorrigibility slopes remain flat or decrease across corpora (e.g., Incorrigibility drops by $-0.39$ percentage points per order of magnitude of compute in Nemotron-CC).

Limitations and Future Work

  • Lack of Formal Theory: The framework is empirical and does not yet provide a closed-form theoretical proof linking data mixtures to the relative exponent $\Delta\beta$.
  • Perplexity vs. Real-World Utility: While bits-per-byte (BPB) shows clear linguistic divergence, the exact relationship between small raw language modeling loss gaps and downstream user utility remains open.
  • Scope: Evaluations focused solely on pretraining across three domains; future work must evaluate whether targeted data augmentation can reverse negative exponents, and examine how relative trajectories evolve under multimodal architectures and post-training (RLHF/instruction tuning).

Summary updated

Abstract

Scaling laws describe how language models improve with additional data, parameters, and compute. While widely used, they are typically measured on aggregate test sets. Aggregate evaluations yield clean trends but average over heterogeneous subpopulations, obscuring performance disparities. We introduce relative scaling laws, which track how performance gaps between test distributions evolve with scale rather than focusing solely on absolute error. Using 255 decoder-only Transformers trained under matched-compute (IsoFLOP) budgets from $10^{18}$--$10^{20}$ FLOPs on standard pretraining datasets, we find diverse trajectories: academic domains on MMLU converge toward parity; regional English dialects shift depending on population size; and clusters of AI risk behaviours split, with capability- and influence-related risks increasing during pretraining while adversarial risks do not. These results show that although scaling improves overall performance, it is not a universal equalizer. To support further study, we release all model checkpoints from this work to enable practitioners to measure relative alongside traditional scaling laws, in order to better prioritize robustness challenges in light of the bitter lesson
BibTeX
@inproceedings{held2026relative,
  author = {William Barr Held and David Leo Wright Hall and Percy Liang and Diyi Yang},
  title = {{Relative Scaling Laws for LLMs}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=zqmS3yzV85},
}
Figure