Abstract
In modern machine learning, the scarcity of high-quality, task-specific data often necessitates the use of "surrogate" data—samples drawn from synthetic generators, related domains, or different distributions. While surrogate data is abundant, its inherent bias relative to the target task can degrade performance if integrated naively. This paper establishes a theoretical and empirical framework for training models on a mixture of real and surrogate data using Weighted Empirical Risk Minimization (ERM). We introduce a novel scaling law that predicts test error as a function of data volume and weighting, revealing that surrogate data acts as a powerful regularizer. Notably, we demonstrate that even unrelated surrogate data can improve performance, a phenomenon explained through the lens of Stein’s Paradox.
Introduction
The performance of deep learning models is fundamentally tied to the scale and quality of training data. However, acquiring high-quality data from a specific target distribution is often expensive, restricted by privacy, or logistically difficult. To circumvent this, practitioners frequently supplement their datasets with surrogate data.
The challenge lies in the distribution shift: surrogate data is "biased" relative to the target task. Simple Empirical Risk Minimization (ERM), which treats all data points equally, often leads to a "drift" where the model optimizes for the surrogate distribution at the expense of the target performance. This research addresses two critical questions: How can we integrate surrogate data without hurting performance, and can we precisely predict the resulting performance gains?
Methodology: Weighted ERM and Scaling Laws
To balance the trade-off between the volume of surrogate data and its inherent bias, the authors propose Weighted Empirical Risk Minimization (ERM). In this framework, the loss function is modified to assign a weight $\alpha$ to surrogate samples and $(1-\alpha)$ to real samples.
Theoretical Frameworks
The researchers analyzed this approach across four distinct mathematical settings to ensure robustness:
- Gaussian Sequence Models: For non-parametric estimation.
- Sobolev Classes: For function estimation in high-dimensional spaces.
- Low-dimensional Asymptotics: Traditional large-sample statistical theory.
- High-dimensional Ridge Regression: Modern settings where parameters and samples scale proportionally.
The Proposed Scaling Law
The core contribution is a new scaling law that models the expected test error. The law characterizes performance based on the number of real samples ($n$), surrogate samples ($m$), the mixing weight ($\alpha$), and a scaling exponent ($\beta$). This formula allows practitioners to determine the optimal weighting scheme and predict the marginal utility of adding more surrogate data before committing to expensive training runs.
Empirical Validation
The theory was validated across a diverse array of benchmarks, demonstrating the law's universality:
- Natural Language Processing: Sentiment analysis using IMDB reviews as the target (real) data, with Rotten Tomatoes and Goodreads reviews serving as surrogate data.
- Computer Vision: Image classification using CIFAR-10 as real data and CIFAR-100 as surrogate data.
- Medical Research: Survival analysis using the TCGA PanCancer dataset, where gender-specific data served as the distribution shift (e.g., using male patient data as surrogate for female patient predictions).
- Simulated Data: Controlled environments using linear and Gaussian mixture models to precisely manipulate distribution shifts.
Key Insights
Surrogate Data as a Regularizer
A primary finding is that surrogate data does not merely provide more information; it functions as a sophisticated regularizer. By biasing the model toward a stable (though slightly shifted) solution, it reduces the variance of the estimator more than the bias it introduces—provided the weight $\alpha$ is chosen correctly.
The Connection to Stein’s Paradox
The paper provides a theoretical explanation for why even seemingly unrelated surrogate data can be beneficial. According to Stein’s Paradox, when estimating multiple parameters, a combined estimator that "shrinks" individual estimates toward a common point can outperform individual estimators, even if that point is arbitrary. In this context, surrogate data provides the "shrinkage target" that stabilizes the learning process.
The Necessity of Optimal Weighting
The research highlights that unweighted training is risky; as surrogate data volume ($m$) increases, error can actually increase if the bias is high. However, with optimal weighting, the test error is guaranteed to be monotonically decreasing. The authors demonstrate that the optimal $\alpha$ can be effectively tuned using a small validation set.
Conclusion and Future Work
This work provides a rigorous foundation for data augmentation and transfer learning strategies. By treating surrogate data as a weighted regularizer, practitioners can reliably improve model performance across various domains.
Despite its success, the authors note several areas for future exploration. The current scaling law assumes the model can eventually reach the theoretical minimum error (Bayes risk), which may not hold for models with limited capacity. Additionally, while the law assumes a shared scaling exponent ($\beta$) for both data types, future iterations may need to account for significant differences in data complexity. Extending these laws to self-supervised pre-training and reinforcement learning remains a promising frontier for research.
Summary updated