TL;DR
Aggregate scaling laws hide whether performance gaps between subgroups shrink or widen as compute grows. By introducing relative scaling laws—fitting power laws to the error ratios between conditions—this paper demonstrates across 255 compute-controlled models that scale naturally closes gaps in academic domains (MMLU), but exacerbates disparities across global English dialects and selectively amplifies capability/influence AI risks over adversarial ones.
What Problem Does This Paper Address?
Standard neural scaling laws typically show smooth power-law reductions in average loss as models scale up in compute, parameters, and tokens. However, aggregate benchmarks mask disparities across heterogeneous subpopulations:
- Does scaling act as a "universal equalizer" that lifts neglected domains to parity, or does it widen the divide?
- Performance differences at small scales are often confounded by inherent data entropy rather than responsiveness to scale.
Pointwise evaluations at a single scale cannot distinguish whether an observed gap is permanent or destined to vanish under the "bitter lesson" of additional compute.
Methods
The authors formalize relative scaling laws by modeling the ratio of test errors between a treatment distribution and a baseline: $$G(F) = \frac{E_{\text{treatment}}(F)}{E_{\text{baseline}}(F)} = \gamma F^{\Delta\beta}$$ - $\gamma = \alpha_{\text{treatment}}/\alpha_{\text{baseline}}$ captures the initial small-scale disparity. - $\Delta\beta = \beta_{\text{baseline}} - \beta_{\text{treatment}}$ is the relative scaling exponent. - $\Delta\beta < 0$: The treatment improves faster; the performance gap narrows (convergence). - $\Delta\beta > 0$: The treatment improves slower; the gap widens (divergence). - $\Delta\beta = 0$: The gap remains constant.
To ensure rigorous scaling comparisons, the authors train 255 decoder-only Transformers (Qwen 3 architecture) across matched-compute (IsoFLOP) budgets ranging from $10^{18}$ to $10^{20}$ FLOPs using hyperparameter schedules governed by $\mu\text{P}$ principles and AdamW augmented with AdamC and Caution.
To solve the issue of erratic, uncalibrated downstream metrics, the authors introduce a modified MCQ format that evaluates log-probabilities over the full option text (e.g., "A. Nitrogen") rather than bare option letters. This eliminates surface form competition and yields smooth compute-to-loss scaling ($R^2 = 0.61$) that accurately predicts downstream accuracy through a sigmoidal calibration curve.
Modified MCQ Format (Smooth Loss & Predictable Scaling)
┌──────────────────────────────────────────────────────────────┐
│ Question: Which gas is most abundant in Earth's atmosphere? │
│ Prompt Options: A. Nitrogen B. Oxygen C. Argon │
│ Scored Target Tokens: {"A. Nitrogen", "B. Oxygen", ...} │
└──────────────────────────────┬───────────────────────────────┘
│
▼
[ Compute (FLOPs) ] ──(Power Law)──► [ Task Loss ] ──(Sigmoid)──► [ Downstream Accuracy ]
Datasets
- Pretraining Datasets: 85 models were trained on each of three corpora representing different data curation philosophies:
- CommonPile: Openly licensed and public domain text.
- DCLM Baseline: Filtered and deduplicated web-crawl data.
- Nemotron-CC: Hybrid web text and synthetic rephrasings.
- Evaluation Benchmarks:
- MMLU: Academic knowledge split across STEM, Humanities, Social Sciences, and Miscellaneous subjects.
- International Corpus of English (ICE): Written dialects across 10 regions (USA, Canada, Nigeria, Singapore, Sri Lanka, Hong Kong, India, Ireland, Jamaica, Philippines, Trinidad & Tobago).
- Anthropic Advanced AI Risk Evaluations: 154 model-written risk datasets clustered into five high-level categories: Self-Improvement (baseline), Influence, Self-Replication, Scheming, and Incorrigibility.
Key Insights and Ideas
- Scale narrows academic knowledge disparities: Regardless of training data bias, gaps between STEM and other domains (Humanities, Social Sciences) shrink toward parity with increased compute.
- Scale widens dialect disparities based on population: In written English dialects, scaling does not uniformly close gaps. Dialect improvements strongly correlate with the size of each region's online English-speaking population ($R = 0.82\text{--}0.84$). Canada and Singapore track or gain on U.S. English, while Nigeria and Sri Lanka fall further behind.
- Parameter scaling drives dialect divergence, not token scaling: Isolating model parameters vs. training tokens reveals that scaling parameters shifts relative dialect trajectories, whereas scaling data tokens simply shifts all dialects in parallel.
- AI risks split during pretraining: As compute scales, models naturally assign higher probability to behaviors associated with capability and expansion (Self-Improvement, Influence, Self-Replication). In contrast, adversarial behaviors (Scheming, Incorrigibility) stay flat or decline during pretraining.
Key Results
- MMLU Convergence: In CommonPile, Humanities loss starts 29% worse than STEM at $10^{18}$ FLOPs but converges to within 5% at $10^{20}$ FLOPs. On DCLM and Nemotron, Social Sciences and Miscellaneous reach near-parity with STEM.
- Global English Divergence: Every 10× increase in a country’s online speaker population yields a 0.3% to 0.4% improvement in its relative scaling slope against U.S. English ($p < 0.005$). In CommonPile, Singapore begins below Nigeria at $10^{18}$ FLOPs but overtakes it at $6 \times 10^{19}$ FLOPs.
- AI Risk Emergence: Scheming and Incorrigibility slopes remain flat or decrease across corpora (e.g., Incorrigibility drops by $-0.39$ percentage points per order of magnitude of compute in Nemotron-CC).
Limitations and Future Work
- Lack of Formal Theory: The framework is empirical and does not yet provide a closed-form theoretical proof linking data mixtures to the relative exponent $\Delta\beta$.
- Perplexity vs. Real-World Utility: While bits-per-byte (BPB) shows clear linguistic divergence, the exact relationship between small raw language modeling loss gaps and downstream user utility remains open.
- Scope: Evaluations focused solely on pretraining across three domains; future work must evaluate whether targeted data augmentation can reverse negative exponents, and examine how relative trajectories evolve under multimodal architectures and post-training (RLHF/instruction tuning).
Summary updated