Per Class Sample Selection for Noisy Long Tailed Data

61 concepts37 papers1 proposal

When training deep neural networks on real-world datasets, practitioners frequently encounter a "double whammy": label noise (incorrect annotations) and class imbalance (a long-tailed distribution where a few classes dominate and most are rare).

To combat label noise, standard algorithms rely on the small-loss principle. This heuristic assumes that deep networks memorize clean, simple patterns before fitting random noise. Consequently, samples that yield a low training loss are flagged as clean, while high-loss samples are discarded or corrected.

However, when datasets are simultaneously long-tailed, this elegant heuristic breaks down catastrophically.

The Failure Mode: Why Global Thresholds Blindside Minority Classes

A model’s loss on a specific training sample is a function of two distinct properties: the correctness of its label and the model's familiarity with its features. In a long-tailed dataset, the model is repeatedly exposed to majority-class samples, allowing it to quickly become confident and yield low losses for clean data in those categories. Conversely, the model rarely sees minority-class samples. Because of this underrepresentation, the model remains highly uncertain about rare categories early in training.

As a result, perfectly clean samples from rare tail classes naturally produce high training losses.

Standard Global Filtering:
Loss Threshold: -------------[ Global Cutoff ]-------------
Majority Class: [ Clean (Low Loss) ]   |   [ Noisy (High Loss) ]  --> Keeps Clean
Minority Class:                        |   [ Clean (High Loss) ]  --> Accidental Purge!

If an algorithm applies a uniform, global loss threshold across the entire dataset to filter out noise, it creates an unintended bias. As noted by Chuang Zhu et al. in their work on Optimal Transport Learning from Noisy Labels, the small-loss principle fails to distinguish between actual label noise and clean but underrepresented tail samples.

This systematic exclusion creates a destructive feedback loop. Xiaobo Xia et al. (Sample Selection with Uncertainty of Losses) point out that filtering solely by point-estimation losses discards critical, hard examples that are essential for minority-class generalization. By mistakenly purging these rare but valid data points, standard noise-filtering mechanisms further starve the model of minority-class exposure, causing performance on the long tail to collapse completely (Jing Jiang et al., Boosting Facial Expression Recognition by A Semi-Supervised Progressive Teacher).

The Solution: Per-Class Noise Modeling

To resolve this conflict, the core mechanism of recent research shifts from global dataset-level filtering to per-class sample selection. The intuition is simple: we must evaluate how anomalous a sample's loss is relative to its own class baseline, rather than relative to the entire dataset.

Yingsong Huang et al. (Uncertainty-Aware Learning against Label Noise on Imbalanced Datasets) demonstrate that loss values are fundamentally relative to class frequency. A high-loss sample in a minority class may actually be a highly reliable, clean data point, whereas a medium-loss sample in a dominant majority class might be noise. By fitting probability distributions to loss values independently for each class, algorithms can establish dynamic, class-specific baselines that protect rare categories from premature exclusion (Lefan Zhang et al., Learning from Long-Tailed Noisy Data with Sample Selection and Balanced Loss).

Normalizing the Loss Landscape

Instead of altering the thresholding mechanism itself, an alternative way to build intuition for this solution is to look at how we can mathematically normalize the loss landscape before filtering occurs.

The authors of Robust Logit Adjustment for Learning with Long-Tailed Noisy Data (MingCai Chen et al.) achieve this by applying an inverse class frequency re-weighting to loss values prior to noise detection. This scaling artificially suppresses the baseline loss inflation caused by class rarity, effectively decoupling the inherent difficulty of learning an underrepresented class from actual label corruption.

Similarly, Jiaqi Bai et al. (Towards Real-world Scenario: Imbalanced New Intent Discovery) utilize distribution-aware regularization to dynamically lower confidence thresholds for tail classes, ensuring that the model's natural uncertainty about rare concepts does not result in their deletion.

Enforcing Class Balance via Uniform Selection

Another facet of this research direction bypasses loss modeling entirely by enforcing strict class balance during the sample selection step.

Ajmal Mian (UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning) demonstrates that maintaining class balance during selection is often more critical for downstream model performance than the absolute precision of the noise filter. By selecting an equal volume of low-loss samples independently within each class, the training process guarantees that hard or rare categories remain represented in every mini-batch.

This uniform selection strategy prevents the model from developing severe majority-class biases (Kuan-An Su et al., Let the noise flow away: combating noisy labels using normalizing flows). It ensures that clean samples are drawn fairly across the entire label space rather than prioritizing only the easiest-to-learn majority patterns (Peng-Fei Zhang et al., Effective and Robust Adversarial Training against Data and Label Corruptions).

Asymmetric Filtering Strictness

Finally, this intuition can be extended to an asymmetric filtering strategy where the strictness of the noise filter scales with class abundance.

When dealing with highly imbalanced real-world data, Olympia Gounari et al. (FILTERING LPIS DATA FOR BUILDING TRUSTWORTHY TRAINING DATASETS) show that applying strict statistical thresholds to dominant classes while using relaxed limits for rare classes optimizes data quality. This asymmetry aggressively purges low-quality samples from high-frequency categories where data is cheap, while preserving precious training examples in the long tail where every sample is vital for generalization.

Key Foundational Work

Active Learning with Efficient Feature Weighting Methods for Improving Data Quality and Classification Accuracy (2014) — contributed 1 concept elements to this direction.
Iterative Metric Learning for Imbalance Data Classification (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_per_class_sample_selection_77e6a4e4,
  title = {per-class sample selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/per-class-sample-selection/rdr_77e6a4e4d16146ec9bcf380f5c7db3a6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_per_class_sample_selection_77e6a4e4_v1,
  title = {per-class sample selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/per-class-sample-selection/rdr_77e6a4e4d16146ec9bcf380f5c7db3a6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake