The Surprising Effectiveness of Random Data Selection

This direction is split into other directions: random sampling baseline. Its last version stays readable and citable here.

Version 1 of 142 papers50 conceptsfrozen 2026-10-04

In machine learning, we often assume that "smarter" is better. When faced with massive datasets, the natural instinct is to design sophisticated algorithms—active learning acquisition functions, coreset selection heuristics, or influence-based filtering—to identify the most "informative" samples.

However, a growing body of evidence across diverse domains suggests a counterintuitive reality: simple uniform random sampling is remarkably difficult to beat. From large language model (LLM) fine-tuning to kernel methods and medical imaging, researchers are finding that the overhead of complex data selection often outweighs its marginal benefits, and in many cases, random selection is actually the superior strategy.

The Selection Paradox in Modern LLMs

The most striking evidence for the power of random sampling comes from the frontier of Large Language Models. In the context of In-Context Learning (ICL), where a model learns from a few demonstrations in its prompt, Uri Berger and Tal Baumel found that sophisticated strategies for picking demonstrations rarely outperform simple random selection. In their study, In-Context Learning on a Budget: A Case Study in Named Entity Recognition, they demonstrated that labeling just 200 random samples can achieve 90% of the performance of a model trained on a full gold-standard dataset.

This pattern repeats in instruction fine-tuning. Daphne Ippolito and Harshita Diddee conducted an extensive "reality check" in Chasing Random: Instruction Selection Strategies Fail to Generalize. They discovered that automated selection strategies often fail to generalize across different datasets and budgets. Crucially, they noted that the computational and financial cost of running these complex selection algorithms can actually exceed the cost of simply training on the entire dataset.

Theoretical Optimality of Stochastic Selection

The effectiveness of random sampling isn't just an empirical fluke; it has deep theoretical roots. In the realm of classical learning theory, Kasper Green Larsen proved in Bagging is an Optimal PAC Learner that bagging—a method based on simple random sampling with replacement—is a theoretically optimal learner. This suggests that structured data partitioning is often unnecessary for achieving the best possible sample complexity.

Similarly, in kernel methods, Alessandro Rudi and Lorenzo Rosasco showed in Less is More: Nyström Computational Regularization that Nyström subsampling acts as a form of computational regularization. By choosing a specific number of random points, one can achieve optimal learning accuracy while breaking the $O(n^2)$ bottleneck of kernel methods. Their work implies that for many large-scale tasks, the added complexity of "leverage score" sampling provides no statistical advantage over uniform sampling.

Why Random Sampling Works: Diversity and Consensus

Why does a "naive" approach perform so well against mathematically rigorous selection criteria? The research suggests two primary mechanisms:

  1. Preservation of the Data Distribution: Complex heuristics often prioritize "hard" examples or outliers. While these points are informative, they can bias the model away from the underlying data distribution. Ren-Biao Liu and colleagues found in Random Selection Reveals Implicit Knowledge Consensus in Code Generation that random sampling is better at capturing the "implicit consensus" of common patterns, which is more useful for model alignment than focusing on boundary cases.
  2. Robustness to Noise: Sophisticated methods like gradient matching or sensitivity-based sampling are often highly sensitive to noise in the data. Peng-Ling Wu demonstrated in Efficient quaternion CUR method for low-rank approximation to quaternion matrix that uniform sampling can be more robust than probability-based methods when data is noisy, as it doesn't over-index on potentially corrupted "high-importance" points.

Reality Checks Across the Field

The "reality check" has become a recurring theme in recent literature.

Conclusion: The Cost of Complexity

The recurring lesson from these 41 papers is that the "computational tax" of data selection is often too high. Whether it is Antoine Simoulin using random token selection to slash GPU memory usage in Memory-Efficient Fine-Tuning of Transformers via Token Selection or Jose Oramas using random subsets for model interpretation, the conclusion is the same: Random sampling is not just a baseline; it is a powerful, robust, and computationally efficient strategy that often defines the state-of-the-art.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_surprising_effectiveness_of_random_d_c2f005e2,
  title = {The Surprising Effectiveness of Random Data Selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-surprising-effectiveness-of-random-data-selection/rdr_c2f005e2d17441dd9fb9f046535f9c76},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_surprising_effectiveness_of_random_d_c2f005e2_v1,
  title = {The Surprising Effectiveness of Random Data Selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-surprising-effectiveness-of-random-data-selection/rdr_c2f005e2d17441dd9fb9f046535f9c76/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}