Preserving Minority Classes in Dataset Pruning

This direction is split into other directions: minority class preservation. Its last version stays readable and citable here.

Version 1 of 135 papers54 conceptsfrozen 2026-10-04

Neural scaling laws suggest that model performance improves with more data, but the cost of training on massive datasets is becoming unsustainable. Data pruning—the process of identifying and removing redundant or uninformative samples—promises to break these power laws, achieving better accuracy with less compute. However, a recurring challenge across recent research is that standard pruning metrics often inadvertently "erase" minority classes or difficult categories, leading to models that are efficient but fundamentally biased.

The Failure of Global Thresholds

Most data pruning algorithms rank samples by a "difficulty" or "utility" metric and discard the bottom percentage. For example, Ben Sorscher and colleagues in Beyond neural scaling laws: beating power law scaling via data pruning demonstrate that keeping only "hard" samples can accelerate learning.

The problem is that difficulty is rarely distributed uniformly across a dataset. If an entire category of data (e.g., a rare medical condition or a specific lighting condition in video) is inherently more difficult for a model to learn than the majority classes, a global difficulty threshold will disproportionately target that category for removal. Elisa Tsai and Atul Prakash show that this can lead to the total loss of minority classes, a phenomenon they address in Class-Proportional Coreset Selection for Difficulty-Separable Data. When a pruning algorithm targets a specific difficulty range globally, it risks discarding the entirety of a rare but critical category that happens to fall within that range.

The Bias in Data Valuation

Even sophisticated game-theoretic approaches like Data Shapley, which assign a "price tag" to each data point based on its contribution to model accuracy, suffer from this structural bias. Amirata Ghorbani and James Y. Zou found in Data Valuation for Medical Imaging Using Shapley Value that while these metrics are excellent at identifying mislabeled data, they can systematically undervalue high-quality samples from majority classes or over-prioritize minority samples to the point of instability.

Furthermore, Keziah Naggita and Julienne LaChance argue in A case for data valuation transparency via DValCards that many valuation algorithms prioritize points that contribute most to overall model accuracy. In imbalanced datasets, this creates a feedback loop where the majority class is seen as more "valuable" because it drives the bulk of the accuracy metric, leading to the systematic removal of minority class samples during data cleaning.

Stratified Pruning and Error Quotas

To solve this, researchers are moving toward "class-aware" or "stratified" pruning strategies. The core intuition is simple: instead of pruning the dataset as a single pool, we should prune each class independently to maintain the original distribution.

  1. Class-Proportional Selection: Elisa Tsai proposes allocating a fixed pruning budget to each class proportional to its original representation. By performing selection logic within each category, the method prevents the accidental removal of rare classes that might otherwise be discarded by global filters.
  2. Error-Based Quotas: In DRoP: Distributionally Robust Data Pruning, Artem M. Vysogorets and Julia Kempe introduce a method that uses class-specific error rates to set retention quotas. By retaining a higher percentage of samples for classes where the model currently struggles (high recall error), they ensure the training process maintains a balanced signal across the entire distribution.
  3. Prototypical Graph Pruning: For non-Euclidean data, Guibin Zhang and colleagues developed GDeR (GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning), which explicitly protects underrepresented clusters in graph neural networks to prevent the systematic exclusion of minority samples.

Maintaining Structural Integrity

The shift toward class-aware pruning is not just about fairness; it is about maintaining the structural integrity of the data. Fan Li and colleagues demonstrate in Overlapping oriented imbalanced ensemble learning that distributing majority class samples across subsets, rather than aggressive deletion, preserves critical boundary information. Similarly, the YOCO (You Only Condense Once) framework by Yang He emphasizes that maintaining strict class balance is essential when pruning condensed datasets for edge deployment, as low-sample regimes are hyper-sensitive to distribution shifts.

By shifting the focus from "which samples are globally best" to "which samples best represent each category," these new algorithms allow for massive reductions in dataset size—sometimes up to 99%—without sacrificing the model's ability to generalize to the rare and difficult cases that define real-world performance.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_preserving_minority_classes_in_dataset_p_a08ed774,
  title = {Preserving Minority Classes in Dataset Pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preserving-minority-classes-in-dataset-pruning/rdr_a08ed7748c4940518841a379096305e4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_preserving_minority_classes_in_dataset_p_a08ed774_v1,
  title = {Preserving Minority Classes in Dataset Pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preserving-minority-classes-in-dataset-pruning/rdr_a08ed7748c4940518841a379096305e4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}