In the early stages of training a model or evaluating a benchmark, every data point feels precious. Researchers have spent decades developing sophisticated "selection strategies"—algorithms that use uncertainty, diversity, or gradient matching to pick the perfect subset of data. The intuition is simple: if we can only afford a few examples, they should be the most informative ones.
However, a recurring pattern across modern machine learning research reveals a sobering reality: the advantage of these sophisticated strategies is a wasting asset. As the volume of data increases, the performance gap between the most complex selection algorithm and simple random sampling tends to collapse.
The Low-Data Mirage
In extreme low-resource settings, selection strategies are undeniably powerful. When you only have a "support set" of five or ten examples, picking representative templates can be the difference between a functional model and a random guess. Joaquin Vanschoren and Branislav Pecher demonstrate this in Automatic Combination of Sample Selection Strategies for Few-Shot Learning, showing that strategic selection provides a massive head start.
Similarly, in medical imaging, Quan Quan and S Kevin Zhou found that their Sample Choosing Policy significantly reduces error in one-shot or five-shot landmark detection. But even here, the trend is clear: as the number of labeled examples grows, the variance caused by the specific choice of samples vanishes.
The Saturation Threshold
Across different domains, researchers have identified specific "saturation points" where random sampling catches up to curated selection.
- Micro-benchmarking: For evaluating LLMs on tiny test sets, Gregory Yauney and Swabha Swayamdipta show in How Reliable is Language Model Micro-Benchmarking? that specialized selection is vital for sets under 100 examples. However, once a benchmark reaches roughly 250 examples, random sampling becomes just as reliable.
- Few-Shot Learning: Vanschoren et al. found that the marginal utility of complex selection strategies drops significantly once a support set exceeds 30 to 40 examples.
- Active Learning: In the context of fine-tuning segmentation models for disaster relief, Adrianna Janik and Kris Sankaran observed that smart sampling is most effective when labeling 10% to 40% of the data pool; beyond that, the model converges regardless of the strategy.
Why Randomness Wins at Scale
The reason for this convergence is statistical. A small random sample is likely to miss rare but important parts of the data distribution (the "long tail"). Selection strategies work by explicitly seeking out these missing pieces to ensure coverage.
However, as the sample size grows, a random sample naturally begins to cover the entire data manifold. Samarth Sinha and Yoshua Bengio explored this in Small-GAN, noting that while "Core-set" selection helps mimic large-batch training, large random mini-batches eventually provide adequate coverage on their own.
In the era of million-token context windows, this effect is even more pronounced. Jinheon Baek and Sun Jae Lee argue in Revisiting In-Context Learning with Long Context Language Models that for "many-shot" learning, the bottleneck is no longer picking the best examples, but simply finding enough examples to fill the window. When you can fit 1,000 demonstrations into a prompt, a random slice of the training set is likely to be as representative as a carefully retrieved one.
The Hidden Cost of Being Smart
If selection strategies eventually perform the same as random sampling, we must consider the computational overhead. Daphne Ippolito and Harshita Diddee highlight a provocative finding in Chasing Random: Instruction Selection Strategies Fail to Generalize: the cost of running a complex selection algorithm can sometimes exceed the cost of simply training on the entire dataset.
This suggests a "No Free Lunch" theorem for data selection. As Nihal V. Nayak and David Alvarez-Melis conclude in A Critical Look at Targeted Instruction Selection, these algorithms are most valuable in strictly resource-constrained scenarios. In the "large-scale" regime—where we have the compute to process more data—the simplest strategy is often the most robust: just use more data, and let randomness handle the rest.