Stratified Data Pruning for Class Balanced Learning

31 concepts22 papers1 proposal

Data pruning aims to reduce the computational cost of training by identifying and removing redundant or noisy examples. However, when applied to datasets with uneven class distributions, standard pruning metrics often exhibit a "majority bias." By ranking data importance globally, these methods frequently misinterpret rare minority samples as outliers or noise, leading to their accidental removal.

Recent research focuses on stratified data pruning: a shift from global importance ranking to class-aware selection strategies that preserve the structural integrity and fairness of the training set.

The Failure of Global Importance Metrics

Most data pruning algorithms rely on a single score—such as loss, gradient magnitude, or uncertainty—to determine which samples to keep. While effective for homogeneous data, Anand et al. demonstrate that global ranking in multi-domain systems leads to the disproportionate pruning of critical examples from niche tasks or smaller domains.

This occurs because minority samples often occupy "hard" regions of the feature space. As Lyu and Tsang note, curriculum-based selection strategies may mistake these rare but legitimate samples for corrupted labels or noise. Similarly, Joly and Salmon argue that high pruning thresholds for ambiguous data can bias models against minority classes, as the inherent difficulty of learning rare features is misinterpreted as label ambiguity. Even traditional data valuation methods like Shapley values can systematically target minority samples for removal, as shown by Mesana et al..

Preserving Distribution via Label Matching

To counter this bias, researchers have proposed explicit constraints to maintain the original class distribution during the pruning process. Du et al. argue that label matching—ensuring the pruned subset retains the same class ratios as the full dataset—is critical for preventing performance degradation on minority classes. Without these constraints, even advanced selection algorithms may optimize for overall diversity at the expense of categorical integrity.

A common implementation of this idea is the use of class-specific quotas. Vysogorets et al. introduced Distributionally Robust Pruning (DRoP), which allocates data retention quotas based on class-specific error rates. By retaining a higher percentage of samples for "hard" classes with high recall errors, the authors show that pruning can actually improve worst-class accuracy compared to training on the full dataset.

Hybrid and Stratified Selection Strategies

Beyond simple quotas, more sophisticated frameworks use hybrid sampling to balance global informativeness with local class representation. The SIMS framework, proposed by Grosz et al., combines a fixed percentage of guaranteed per-class samples with a larger global pool of high-importance samples. This ensures that rare categories are never entirely eliminated, even at aggressive pruning ratios.

In the context of synthetic data, Tang et al. apply a similar logic by performing pruning within stratified groups based on object categories. This prevents the selection process from favoring common classes or simple scenes, ensuring the resulting dataset remains representative of complex, rare object types.

Structural and Prototypical Preservation

For complex data structures like graphs, Zhang et al. introduced GDeR, which uses prototypical graph pruning to protect underrepresented clusters. By explicitly safeguarding these prototypes, the model maintains performance on imbalanced datasets that standard importance sampling would otherwise degrade.

The intuition across these works is that data "value" is not an absolute, global property. As Liu and Moore observed in early work on non-parametric classification, the efficiency of pruning is deeply tied to class distribution. Modern stratified pruning acknowledges this by treating the dataset not as a monolithic pool, but as a collection of distinct distributions that must each be represented to ensure a model generalizes fairly across the entire tail.

Key Foundational Work

Breaking SVM Complexity with Cross-Training (2004) — contributed 1 concept elements to this direction.
New Algorithms for Efficient High Dimensional Non-parametric Classification (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_minority_class_preservation_fd943e42,
  title = {minority class preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minority-class-preservation/rdr_fd943e422a7243bf9876c5992174220b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_minority_class_preservation_fd943e42_v1,
  title = {minority class preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minority-class-preservation/rdr_fd943e422a7243bf9876c5992174220b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake