TL;DR
Standard scaling laws assume that adding more training data invariably improves or maintains model performance. This paper shows that this assumption breaks down in multi-group settings: adding training data from certain sources can actually degrade performance on specific evaluation subgroups—a phenomenon the authors formalize as negative data externalities. Understanding these data-model tensions provides a concrete lower bound on achievable model improvements and demonstrates that bigger models do not automatically eliminate data inefficiencies.
What Problem Does This Paper Address?
In trustworthy and responsible machine learning, practitioners often focus on algorithmic mitigations or collecting as much data as possible. However, the machine learning community rarely interrogates how training data allocations across sub-populations interact with model capacity.
Conventional scaling laws typically assume performance is monotonic with dataset size. Yet, when datasets comprise diverse, heterogeneous groups, data collected from one sub-population can negatively affect another. The authors formalize and investigate this "data inefficiency," exploring when, why, and how adding training data from one group hurts predictions on another.
Methods
- Formalizing Negative Data Externalities: The authors define a negative data externality as occurring when the expected risk on an evaluation group $g_{\text{eval}}$ increases after adding randomly sampled training points from any source group.
- Lower-Bounding Model Improvement (Claim 1): The authors prove that the maximum externality magnitude ($\Delta_{g_{\text{eval}}}$) establishes a lower bound on potential performance gains on that group—achievable simply by sub-sampling or partitioning training sets—without harming disjoint groups.
- Diagnosing the Mechanisms: Using synthetic affine models, they delineate two distinct root causes:
- Model capacity limits / misspecification: Groups have different optimal parameters (e.g., different true intercepts). Adding data from another group pulls the shared model away from the subgroup's optimum.
- Data distribution & noise differences: The true underlying model is identical across groups, but feature distributions or noise scales differ. Adding low signal-to-noise out-group points introduces harmful variance or bias until out-group sample size becomes massive.
Datasets
The authors evaluate data externalities using two established benchmark datasets (no new benchmarks were created):
- Goodreads Book Reviews: A binary classification task predicting whether a review gives 5 stars versus fewer, using 1,000-dimensional TF-IDF vectors. The groups are defined by book genres: history/biography (200,000 train instances) and fantasy/paranormal (200,000 train instances).
- CivilComments (with identities): A text toxicity classification task on online comments. The authors analyze comments associated with the four largest demographic identity groups: female (53,429 train), male (44,848 train), Christian (40,423 train), and Muslim (21,006 train).
Key Insights and Ideas
- The "Saturation" Threshold: Negative externalities primarily occur once the target group has sufficient representation in the training set. At this saturation point, the variance-reduction benefits of pooling extra out-group data are outweighed by the bias introduced toward the out-group distribution.
- Model Interventions Depend on the Root Cause: If externalities stem from capacity bottlenecks, expanding model capacity in a targeted manner (e.g., group-specific parameters) works best. If caused by noise imbalances, splitting models discards potential cross-group transfer; sample re-weighting is more appropriate.
- More Capacity Does Not Guarantee Immunity: A common intuition is that larger neural networks mitigate cross-group interference. Surprisingly, increasing model depth can actually exacerbate the magnitude of negative externalities on certain subgroups.
Key Results
- Goodreads (Lasso Regression): When holding the target genre's training samples fixed at moderate-to-large sizes (e.g., 25,000–200,000 reviews) and incrementally adding samples from the other genre, per-group AUROC steadily decreases. Sub-sampling the training data yields higher per-group AUROC on both genres compared to training on the pooled dataset ($n = 400,000$).
- CivilComments (Miniature BERTs): The authors fine-tuned compact BERT variants varying in depth ($L \in \{2, 4, 8\}$ layers) and hidden dimension ($H \in \{128, 256, 512\}$).
- Adding out-group data caused noticeable AUROC drops on identity groups (up to $\sim 0.005$ to $0.010$ AUROC decrease).
- For the female and Muslim subgroups, deeper models ($L=4, 8$ at $H=128$) suffered larger negative externalities than shallower models ($L=2$), demonstrating that greater capacity can amplify out-group interference.
Limitations and Future Work
- Known Group Attributes: The framework assumes discrete, predefined group labels at both training and test time. Identifying externalities across overlapping, continuous, or unannotated latent groups remains an open challenge.
- Computational Scalability: Selectively subsetting training datasets or deploying per-group models becomes computationally intractable as the number of demographic subgroups grows.
- Fixed Fine-Tuning Schedules: Hyperparameters (learning rates, batch sizes) were held static across model variants; future work could analyze whether dynamic per-group hyperparameter tuning alleviates these tensions.
- Algorithmic Design: The authors call for developing learning algorithms that guarantee data efficiency and bounded externalities from the outset, aligning traditional accuracy goals with data minimization and algorithmic fairness.
Summary updated