The dominant paradigm in modern machine learning has long been "more is better." Scaling laws regularly predict that inflating dataset sizes by orders of magnitude will yield predictable, continuous improvements in model performance. However, across domains ranging from large language models to computer vision and specialized regression systems, researchers are increasingly hitting a wall: the data scaling plateau.
Understanding why data scaling encounters diminishing returns requires moving past the assumption that datasets are homogeneous engines of knowledge. Instead, data utility follows a heavily concave trajectory, where massive volume often introduces informational redundancy, superficial pattern copying, and disproportionate computational costs.
The Concave Nature of Data Utility
To understand why adding more data eventually stops helping, it is useful to look at how data translates into model utility. In the early stages of training, every new sample provides novel information, mapping out undiscovered regions of the problem space. But as the dataset grows, the probability that a new sample contains genuinely novel features drops exponentially.
This phenomenon can be formalized mathematically. In the context of commercial data procurement and IoT systems, Mohammad Abu Alsheikh et al. demonstrate in Optimal Pricing of Internet of Things: A Machine Learning Approach that machine learning service quality behaves as an exponential or logarithmic function of training data volume. Initial data points provide steep accuracy gains, but they quickly reach a saturation point where the financial and computational cost of acquiring additional data far outweighs the marginal performance improvement.
A similar economic and technical framework is proposed by Dusit Niyato et al. in Market Model and Optimal Pricing Scheme of Big Data and Internet of Things (IoT), confirming that this concave utility curve is a fundamental property of data-driven systems. Once a model has learned the underlying distribution of a task, expanding the dataset within that same distribution yields negligible structural updates.
Why More Data Can Hurt: Redundancy and Format Copying
In superficial alignment tasks like instruction tuning for Large Language Models (LLMs), simply scaling data volume can actually degrade performance. This counterintuitive "less is more" reality is driven by two main factors: informational redundancy and the limits of what fine-tuning actually accomplishes.
In What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning, Wei Liu et al. show that model performance during instruction tuning frequently hits a ceiling or declines when excessive data is introduced. They find that the volume of truly effective alignment data is remarkably small; adding samples beyond this threshold introduces harmful redundancy and noise.
The root cause of this saturation lies in what instruction tuning actually does. As Sreyan Ghosh et al. argue in A Closer Look at the Limitations of Instruction Tuning, instruction tuning rarely expands a model's factual knowledge base. Instead, it primarily teaches the model the format of a response (e.g., how to behave like an assistant). Once the model masters this format, scaling the instruction dataset fails to improve factuality. Instead, it forces the model to engage in superficial "pattern copying," which can increase hallucinations and degrade performance.
The Temporal Dynamics of Information Saturation
The diminishing returns of data scaling are not just a function of dataset design; they are deeply tied to the mechanics of how neural networks optimize over time. Data points do not exert a uniform influence throughout the training process.
In Capturing the Temporal Dependence of Training Data Influence, Jiachen T. Wang et al. analyze how a sample's impact shifts depending on when it is introduced during training. They discover that during the middle phase of training, models enter a state of "influence saturation."
During this period, new data points that closely resemble previously processed examples contribute almost nothing to the final model state. This is not because the model is forgetting prior knowledge, but because its internal representations have already adapted to those features, creating a sharp drop in the marginal information gain of redundant data.
Circumventing the Plateau: Heterogeneity Over Volume
If raw volume inevitably triggers saturation, how can practitioners continue to drive model progress? The consensus across recent literature points away from data hoarding and toward data curation, prioritizing heterogeneity and targeted difficulty over sheer scale.
The power of data diversity is starkly illustrated in compact language modeling. Louis Martin et al. demonstrate in Les modèles de langue contextuels Camembert pour le français : impact de la taille et de l'hétérogénéité des données d'entrainement that pre-training a model on just 4GB of highly diverse, web-crawled text can match the performance of an identical architecture trained on 138GB of less varied data. For models with moderate parameter counts, expanding a homogeneous corpus yields diminishing returns, whereas increasing data heterogeneity directly improves downstream generalization.
Similarly, Vincent Micheli et al. show in On the importance of pre-training data volume for compact language models that downstream task performance for compact models plateaus rapidly. Increasing a training corpus forty-fold can result in negligible accuracy gains if the data does not introduce new structural concepts.
To break through these plateaus, data collection must shift from random representative sampling to active, model-centric selection. As Peibei Cao et al. explain in Image Quality Assessment: Integrating Model-centric and Data-centric Approaches, massive datasets naturally become "easy" for models because standard sampling methods over-represent common patterns. To force progress, datasets must be curated using "failure predictors" that specifically target a model's current blind spots.
When volume alone is pursued, systems inevitably stall. To overcome the scaling plateau, data must be treated not as a uniform commodity, but as a finite resource where diversity, complexity, and information density dictate ultimate utility.