Data Pruning for Efficient Large Scale Model Training

55 concepts53 papers1 proposal

Modern deep learning is often a battle of attrition against massive datasets. As models grow, the computational and financial costs of processing every available data point have become unsustainable. However, research into informative subset selection—often called data pruning or coreset selection—suggests that much of this data is redundant. By identifying and training on only the most "potent" samples, we can achieve comparable or even superior performance with a fraction of the resources.

The Geometry of Information

The fundamental intuition behind subset selection is that a dataset’s utility is not distributed uniformly. A small number of samples often capture the underlying geometry and decision boundaries of the entire distribution.

To capture this geometry, Axiotis et al. propose using $k$-means clustering and sensitivity sampling to pick representative points that approximate the training signal of the full collection. Similarly, Xiao and Shan use Optimal Transport to ensure that the feature distribution of a selected "coreset" perfectly mirrors the original dataset. By matching the statistical properties of the whole, these methods allow models to generalize to unseen data even when trained on as little as 10% of the original volume.

The Difficulty Paradox

A recurring question in this field is: which samples are the most informative? Intuitively, one might pick "hard" samples that the model struggles with, or "easy" samples that represent the core patterns.

However, the authors of Moderate Coreset argue that focusing exclusively on extremes is a mistake. They show that selecting data points with scores near the median of the distribution creates a more robust proxy for the full dataset, especially in the presence of label noise. Grosz et al. further refine this by proposing an adaptive strategy: keeping "hard" samples when data is plentiful to push the model's limits, but shifting to "easy" representative samples when the pruning budget is tight to ensure the model learns the basic structure first.

Learning How to Select

The most significant bottleneck in data selection is the "selection tax"—the computational cost of determining which samples are valuable. If you have to train a model to figure out what data to use, you haven't saved much time.

Recent work focuses on making selection "training-free" or transferable:

  • Transferable Selection: Jain et al. introduced SubSelNet, a framework that learns to predict the value of data points for unseen architectures, allowing researchers to reuse selection logic across different models.
  • In-Context Selection: Wang et al. developed Data Whisperer, which uses few-shot in-context learning to identify informative samples for Large Language Model (LLM) fine-tuning without any additional training, achieving 35x speedups over previous methods.
  • Sequential Decision Making: Jha et al. treat selection as a reinforcement learning problem, training a policy to pick the most impactful 5% of a dataset, which can actually outperform training on the full 100% by filtering out noise.

Beyond Samples to Tokens and Trajectories

Selection is also moving beyond simple image-by-image or document-by-document filtering. In the realm of instruction tuning, Fu et al. demonstrate that focusing on specific "potent" tokens within a response is more effective than selecting entire samples.

Furthermore, the way a model's loss changes over time—its "trajectory"—provides a rich signal for selection. Nagaraj et al. show that by measuring how closely a training sample’s loss trajectory aligns with a validation set, we can identify the most impactful data for generalization without the massive overhead of calculating gradients for every point.

By treating data as a strategic resource to be curated rather than a commodity to be consumed in bulk, these methods make large-scale machine learning more accessible, sustainable, and robust.

Key Foundational Work

Gaussian Processes for Bayesian Classification via Hybrid Monte Carlo (1996) — contributed 1 concept elements to this direction.
An Information Theoretic Approach to Detection of Minority Subsets (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_informative_subset_selection_aa271aeb,
  title = {informative subset selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/informative-subset-selection/rdr_aa271aeb2d114027ba5c8f974ca829ab},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_informative_subset_selection_aa271aeb_v1,
  title = {informative subset selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/informative-subset-selection/rdr_aa271aeb2d114027ba5c8f974ca829ab/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake