Deep learning's appetite for data has led to the creation of massive, web-scale datasets. However, a fundamental tension exists within this paradigm: as a dataset scales up in raw volume, its informational density scales down. In Learning and Data Selection in Big Datasets, Hadi G. Ghauch demonstrates that the size of a sufficient dataset required to maintain a target model accuracy grows sub-linearly relative to the total available data.
This sub-linear growth implies that large-scale datasets are heavily saturated with informational redundancy. Training on these repetitive samples wastes hardware resources, slows down convergence, and can even degrade model generalization. To bypass this computational wall, researchers are shifting from brute-force data collection to algorithmic data curation: identifying a minimal, highly informative "core subset" that preserves or exceeds the utility of the full dataset.
The Mathematics of Diminishing Returns: Submodularity
To systematically eliminate redundancy, we need a mathematical framework that formalizes the concept of "diminishing returns." This is precisely the domain of submodular functions. A set function is submodular if the marginal benefit of adding an element to a set decreases as the size of the set increases.
In the context of data selection, if you already possess a diverse, representative set of training examples, adding another highly similar example yields negligible marginal utility. Jacob M. Schreiber et al. developed the apricot library to scale submodular optimization to massive datasets, providing a principled alternative to random sampling.
Why does submodularity map so cleanly to machine learning datasets? Kai Wei et al. proved that the data likelihoods for common classifiers (such as Naïve Bayes and Nearest Neighbor) are naturally submodular functions. By maximizing these functions, frameworks like Filtered Active Submodular Selection (FASS) choose subsets that are simultaneously informative (high uncertainty) and representative (high coverage).
Rather than relying on manual, "vibe-based" heuristics to mix instruction-tuning data, H. S. V. N. S. Kowndinya Renduchintala et al. introduced SMART, a framework that uses submodular functions to automatically select non-redundant instruction mixtures. Their work proves that a strategically pruned subset of data is often more effective than a massive, uncurated "kitchen sink" approach.
Reading the Model's Mind: Training Dynamics
While submodular optimization over raw features is powerful, calculating pairwise similarities across millions of high-dimensional samples can become a computational bottleneck itself. An alternative approach is to let the model tell us which data is redundant by observing its training dynamics.
If a model perfectly predicts a sample early in training and its prediction never wavers, that sample offers no new gradient information. Muyang He leveraged this intuition in Large-scale Dataset Pruning with Dynamic Uncertainty, tracking how a model's predictions fluctuate during early epochs to achieve 25% lossless pruning on ImageNet. Similarly, Joshua Tan utilized the Variance of Gradients (VoG) as an influence score in Influence Scores at Scale for Efficient Language Data Sampling, safely discarding half of the training data in production-grade systems with virtually no loss in accuracy.
[ Raw Dataset ] ---> [ Early Training (1-2 Epochs) ] ---> [ Track Logit Fluctuations ]
|
v
[ Optimized Core Subset ] <--- [ Discard Low-Variance Samples ] <--- [ Compute Influence Scores ]
By mapping out data using these training trajectories—a technique known as data cartography—Cornelia Caragea et al. showed that strategically pairing "easy-to-learn" samples with "ambiguous" ones yields superior accuracy and calibration while using significantly less data than standard random mixing.
Why Less Data Can Mean Better Learning
It is intuitive that pruning data saves compute, but a more surprising result is that removing redundancy often improves model performance.
Redundancy is rarely uniform; it typically manifests as an over-representation of simple, majority-class concepts or environmental biases. For instance, fixed-position surveillance cameras generate massive datasets where the background and lighting remain identical across thousands of frames. The authors of Data Pruning Can Do More: A Comprehensive Data Pruning Approach for Object Re-identification demonstrated that removing these redundant frames speeds up training by 35% while maintaining or improving accuracy, because the model stops wasting capacity on overrepresented backgrounds and focuses on unique structural features.
When a model spends its fixed training budget repeatedly processing identical states, it fails to master the diverse edge cases required for broad generalization. This pattern holds true across diverse domains:
- Reinforcement Learning: Aditya Mohan showed that training an agent on a carefully subselected, representative subset of instances improves generalization on unseen test data compared to training on the full, redundant set.
- Generative Models: Rania Briq et al. discovered that Latent Flow Matching models exhibit extreme stability, generating nearly identical outputs from the same noise seed even when trained on 75% less data.
- Language Modeling: Newsha Ardalani demonstrated that aggressive fuzzy deduplication acts as a compute multiplier, fundamentally reshaping neural scaling laws by accelerating the learning rate per token.
Accelerating Evaluation via Anchor Points
The challenge of informational redundancy extends beyond training; it also creates an evaluation bottleneck. Running massive language or generative models through tens of thousands of benchmark questions is prohibitively expensive and slow.
However, test items within a benchmark are often highly correlated. If an LLM answers a difficult calculus question correctly, it is highly likely to answer a simpler algebra question correctly. Kawin Ethayarajh et al. formalized this via "Anchor Points," demonstrating that model confidences are tightly correlated across architectures. This allows researchers to micro-benchmark models and predict failure points across entire datasets using a tiny fraction of the usual evaluation cost.
By treating evaluation as a sparse optimization problem, Taolin Zhang et al. developed SparseEval, which identifies a minimal subset of weighted anchor questions to estimate full benchmark performance with a 5x reduction in samples. Similarly, Felipe Maia Polo et al. introduced tinyBenchmarks, compressing suites like MMLU down to just 100 curated examples while maintaining a less than 2% estimation error. Whether evaluating text-to-image diffusion models (Lin Zhao et al.) or large language models, set-wise optimization proves that we can measure model capabilities far more efficiently by eliminating the redundancy embedded in our benchmarks.