When building large language models (LLMs), practitioners rely heavily on scaling laws to decide how to allocate their computational budgets. For years, the prevailing heuristic assumed that optimal data allocation strategies were scale-invariant. The common practice was simple: find an effective data mixture or a specific token-to-parameter ratio using a small, inexpensive proxy model, and then scale it up linearly to train production-grade models.
However, a growing body of research reveals that this assumption is fundamentally flawed. The optimal composition of training data—whether measured by domain ratios, language distribution, or data quality—is highly scale-dependent. What works beautifully at a small scale can fail at a large scale because different data types saturate at different rates, and data quality dynamically alters the optimal balance between model capacity and token count.
Why Data Mixtures Shift: Saturation and Learning Potential
To understand why data mixtures must change with scale, consider how a model learns from different domains, such as web text, code, and mathematics. At the beginning of training or in small-capacity regimes, a model benefits most from data that yields rapid loss reduction. But as the total compute budget and token count expand, these easy-to-learn domains quickly saturate, offering diminishing returns.
In AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs, Feiyang Kang et al. show that data sources providing rapid loss reduction at smaller scales can experience rapid saturation. As training progresses to larger scales, the optimal strategy must dynamically shift toward more complex domains that possess higher remaining learning potential to ensure balanced downstream capabilities.
This scale-dependency applies across various training paradigms:
- Multilingual Pre-training: Ping Guo et al. demonstrate in Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining that cross-lingual interactions change depending on model capacity and token budget. At smaller scales, the optimal mix prioritizes data that accelerates early convergence, whereas larger scales require a shift toward complex data capable of supporting sustained learning over longer durations.
- Supervised Fine-Tuning (SFT): Even during downstream alignment, the authors of Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models find that the ideal proportions of data domains (such as math vs. general conversation) change significantly based on model size and token budgets. Relying on a static mixture optimized on a small proxy model risks severe performance degradation at production scale.
Breaking the Rule of 20: How Data Quality Reshapes the Frontier
The scale-dependency problem does not just affect what domains you select; it fundamentally alters how much data you should train on relative to model size. Standard heuristics, like the Chinchilla "Rule of 20" tokens per parameter, assume a fixed, universal ratio for compute-optimal training.
Recent work directly challenges this assumption. In Predictable Scale (Part II) --- Farseer: A Refined Scaling Law in LLMs, Houyi Li et al. demonstrate that the optimal token-to-parameter ratio must actually increase as the total compute budget grows. Frontier models require disproportionately more data per parameter than small-scale models to maintain compute efficiency.
Furthermore, this ratio is highly sensitive to data curation. In How Text Quality Interventions Reshape Neural Scaling Laws for LLMs: Empirical Study, Newsha Ardalani et al. empirically prove that text quality interventions—such as deduplication or LLM-guided rewriting—fundamentally reshape the mathematical exponents of neural scaling laws.
When the training corpus is of exceptionally high quality, the compute-optimal frontier shifts: you can train significantly smaller models for much longer durations on massive token counts. Ignoring how data quality alters these scaling coefficients leads to massive inefficiencies, causing practitioners to over-invest in model size when they should be investing in data volume, or vice versa.
Predicting the Shift via Composition-Aware Scaling Laws
If optimal data mixtures and token ratios are moving targets, how can we find them without running prohibitively expensive trial-and-error grid searches at scale? The solution lies in formulating scaling laws that treat data composition as an explicit, joint variable alongside parameter count and compute.
[ Small Scale Proxy ] ──(Traditional Linear Extrapolation)──> [ Suboptimal Large Model ]
│
└──(Composition-Aware Scaling Laws)───────────────> [ Compute-Optimal Large Model ]
• Models domain saturation
• Adjusts token-to-parameter ratios
• Accounts for cross-domain interactions
In Scaling Laws for Optimal Data Mixtures, Mustafa Shukor et al. developed joint scaling formulations that model the explicit interactions between model size, training duration, and domain composition. By capturing how the ideal ratio of data domains shifts as a function of compute, their approach allows practitioners to mathematically extrapolate the optimal large-scale mixture from small-scale runs.
Similarly, Feiyang Kang et al. introduced the projektor framework in Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sources. This framework uses Optimal Transport to measure data relevance from tiny pilot samples while treating scaling law parameters as dependent on the data composition, avoiding the flawed assumption of a uniform scaling coefficient.
In specialized settings like continual pre-training, Haoran Que et al. showed in D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models that incorporating the exact ratio of domain-specific to general data into the scaling law allows accurate performance forecasting at a fraction of the typical computational cost.
Co-Evolving Architectures
Finally, this dynamic view of scaling extends to the neural architectures themselves. Just as data mixtures must adapt to the compute budget, the optimal model architecture must adapt to the data's evolving volume and complexity.
In AdaXpert: Adapting Neural Architecture for Growing Data, Jiaxiang Wu et al. demonstrated that while compact architectures excel when data is scarce, the relationship reverses as data grows. Static architectures cannot maintain peak efficiency across the entire lifecycle of a growing dataset, meaning that compute-optimal data allocation must ultimately co-evolve with dynamic architectural adjustments.
Key Foundational Work
Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sources (2023) — contributed 2 concept elements to this direction.
AdaXpert: Adapting Neural Architecture for Growing Data (2021) — contributed 1 concept elements to this direction.