Abstract
Contrary to the "more data is always better" mantra, adding training data from one source can sometimes decrease a model's performance on specific subgroups. This paper formalizes these "negative data externalities" and demonstrates through experiments on real-world datasets that simply scaling up data or model size doesn't always fix—and can sometimes worsen—these performance tensions.
Introduction
In machine learning, we often assume that more data leads to better models. Standard "scaling laws" typically predict that error decreases monotonically as dataset size increases. However, this paper challenges that assumption by looking at group performance.
The authors identify a phenomenon they call negative data externalities: a situation where adding training data from one source (e.g., one demographic group or topic) actually lowers the predictive performance for a different evaluation group. This is a critical issue for "data efficiency"—the goal of using available data to achieve the best possible performance for all target populations. If adding data hurts some users, our training procedures are inefficient and potentially unfair.
Methods
The authors provide a formal framework to characterize these externalities:
- Formal Definition: They define a negative data externality as a case where the expected risk (loss) for an evaluation group increases when randomly sampled training data is added to the mix.
- Theoretical "Room for Improvement": They prove that the existence of an externality implies a lower bound on how much a model could be improved simply by being smarter about which data it uses.
- Mechanisms of Failure: They use illustrative linear models to show two ways these arise:
-
Distributional Shifts: Even if the underlying "true" model is the same, differences in feature noise or spread between groups can cause temporary performance dips (error spikes) when data is first introduced.
Negative data externalities arising from distributional shifts, where adding data from group B temporarily increases the error for group A. - Capacity Constraints: When a model is too simple to capture different patterns in different groups, data from Group B "pulls" the model away from the optimal fit for Group A, leading to persistent performance degradation.
Negative data externalities due to capacity constraints, where the model cannot simultaneously fit both groups, leading to a persistent increase in error for group A. -
Empirical Testing: They test these ideas using sub-sampling experiments, training models on varying ratios of data from different groups and measuring the impact on per-group AUROC.
Datasets
The paper evaluates its hypotheses on two well-known real-world datasets:
- Goodreads: A book review dataset where the task is to predict if a review is 5 stars. The authors look at "externalities" between genres, specifically History/Biography vs. Fantasy/Paranormal.
- CivilComments (with identity information): A dataset for toxicity classification in online comments. They focus on the four largest identity groups: female, male, Christian, and Muslim.
Key Insights and Ideas
- The Saturation Point: Negative externalities tend to manifest most clearly when a group already has "enough" of its own data. Once the model has "saturated" on Group A's data, adding Group B's data is more likely to introduce bias than to help reduce variance.
- The Model Capacity Paradox: Intuitively, one might think a larger, more complex model would be better at "juggling" diverse data sources. However, the authors found that increasing model complexity (adding layers) can actually exacerbate negative externalities. In some cases, deeper BERT models showed larger performance drops for specific groups when "outside" data was added compared to shallower models.
- Data-Model Synergy: Data efficiency isn't just a property of the dataset; it's about the interaction between the data and the specific modeling choices (architecture, loss function, etc.).
- Fairness and Privacy Links: These externalities have direct implications for fairness (who does the model hurt?) and participation (should a group contribute data if it might degrade the model for others?).
Key Results
- Goodreads: In the book rating task, adding reviews from the "other" genre frequently caused a measurable dip in AUROC for the target genre. For example, if you have 100k History reviews, adding 300k Fantasy reviews can make the model worse at predicting History ratings.
- CivilComments: Negative externalities were pervasive across all four identity groups.
- Miniature BERT Scaling: When fine-tuning BERT models of different sizes on the toxicity task, the authors observed that for the "female" and "Muslim" groups, adding more general data caused a sharper performance decline in deeper models (8 layers) than in shallower ones (2 layers), despite the deeper models having higher overall average performance.
Limitations and Future Work
The authors acknowledge several areas for further exploration:
- Detection: Currently, identifying these externalities requires knowing the groups beforehand and performing expensive sub-sampling experiments. Finding ways to detect them for "unknown" groups is a major challenge.
- Mitigation: While the paper proves that "room for improvement" exists, it doesn't provide a universal algorithm to eliminate externalities. The "best" fix (e.g., reweighting data vs. changing model architecture) depends on the underlying cause.
- Computational Limits: Sub-sampling and training multiple model variations is computationally heavy, making it difficult to scale this analysis to datasets with many overlapping groups.
- Algorithmic Design: A key future goal is to design learning procedures that guarantee no negative data externalities from the outset.
Summary updated