In the world of large-scale machine learning, "scaling laws" have become a reliable compass. These empirical observations dictate that model performance improves log-linearly as we increase data, parameters, or compute. However, traditional scaling laws typically assume an independent and identically distributed (i.i.d.) setting, where scaling up simply means getting "more of the same" data.
In reality, modern machine learning models are trained on a "soup" of diverse data sources with varying quality, relevance, and domain origins. In the paper "Model Performance Scaling with Multiple Data Sources," Tatsunori Hashimoto addresses this practical scenario: If you change the recipe (the mixture proportions of your data), can you still accurately predict how the model will perform?
The Challenge of Mixed Data Sources
Modern systems, such as large language models, are rarely trained on a single monolithic dataset. Instead, they combine web crawls, books, specialized technical papers, and more. This introduces two primary challenges:
- Prediction: If a model is trained on 1,000 samples from Source A and 500 from Source B, how will it perform? What happens if that scales to 10,000 and 5,000?
- Policy: Given a fixed budget for data collection or labeling, what is the optimal ratio of sources to acquire?
Existing scaling laws focus on the total sample size ($n$) but ignore the composition ($q$). This paper bridges that gap by treating model performance prediction across varying data mixtures as a distinct learning problem.
A New Functional Form for Scaling
The core contribution of the paper is a novel functional form for predicting error. The author begins with the standard scaling law equation:
$$\log(\text{error}) \approx -\alpha \log(n) + C$$
The pivotal insight is the conjecture that the slope ($\alpha$) is invariant to data composition, while the composition only affects the intercept ($C$).
To model how the composition $q$ affects the intercept $C$, Hashimoto leverages the theory of V-optimal experimental design. For linear regression, excess loss relates to the trace of the inverse of the information matrix. This theoretical foundation leads to a hypothesized "Rational Function" for the intercept:
$$C_\lambda(q) := \sum_{i=1}^M \left( \sum_{k=1}^K \lambda_{ik} q_k \right)^{-1}$$
Here, $q_k$ represents the fraction of data from source $k$, and $\lambda$ are parameters learned from pilot runs.
The Workflow: 1. Run small "pilot" training runs on subsamples of data with varying sizes ($n$) and mixtures ($q$). 2. Record the resulting errors. 3. Fit the parameters $\alpha$ and $\lambda$ to these observations. 4. Use the fitted model to extrapolate and predict performance for much larger datasets or entirely different compositions.
Key Insights
- The Invariance of Alpha: The "learning rate" (the slope $\alpha$) remains remarkably stable across different data mixtures. This implies that the fundamental difficulty of learning a task is tied to the model architecture and the task itself, while data quality merely shifts the baseline performance.
- Rational > Linear: Practitioners often assume data sources contribute linearly to performance, akin to a weighted average. Hashimoto proves that the relationship is actually inverse and nonlinear. Data sources exhibit diminishing returns, and their complex interactions are best captured by rational functions.
- MLPs Overfit Scaling: When testing a standard Multi-Layer Perceptron (MLP) to predict the scaling curve, the MLP fits the training points perfectly but fails drastically at extrapolation. The "physics-informed" rational function, however, generalizes reliably to data sizes 4-10x larger than the pilot set.
Empirical Results
The approach was rigorously validated across diverse tasks, including Amazon Sentiment Analysis, Task-Oriented Dialogue (SM-CalFlow), Multi-domain Machine Translation, and Multitask Question Answering.
The Rational Function approximation significantly outperformed baselines—such as linear models or simple data-size-only scaling—in extrapolation tasks.
- Amazon Sentiment: The rational function achieved an $r^2 \approx 0.96$ for extrapolation, compared to just $0.76$ for a linear model.
- Machine Translation: In this highly challenging test, many baselines achieved worse-than-random predictions ($r^2 < 0$). The rational function, however, maintained a strong $r^2 \approx 0.83$.
- High Efficiency: The proposed model can accurately predict the performance of a system trained on 4,000 examples using a pilot dataset of only 1,200 examples.
Limitations and Future Directions
While powerful, the method has limitations. Pilot datasets, though relatively small, can still be expensive to run for massive LLMs. Additionally, the theory assumes a fixed model architecture; if model parameters ($N$) are scaled simultaneously with data ($n$), the slope $\alpha$ might change. Finally, prediction accuracy naturally degrades when extrapolating beyond a 10x increase in data size.
Looking forward, this rational function scaling law opens the door for advanced active learning loops, potentially allowing systems to automatically and dynamically decide which data source to sample from next to maximize performance under a budget.
Summary updated