Self-Supervised Feature Learning for Label Noise Resilience

27 concepts20 papers1 proposal

Deep neural networks possess an immense capacity to memorize data. While this expressiveness allows them to learn complex patterns, it becomes a major liability when training datasets contain corrupted, incorrect, or biased annotations—commonly referred to as label noise. In standard supervised learning, feature extraction and label classification are tightly coupled. If a model is forced to minimize cross-entropy loss against corrupted labels, it will warp its internal feature representations to accommodate those errors, degrading its ability to generalize to clean data.

To solve this problem, a powerful paradigm has emerged: decoupling feature learning from label assignment using self-supervised learning (SSL). By forcing a network to learn representations based entirely on the intrinsic structure and statistical regularities of the data itself, the model becomes remarkably resilient to label noise.

Why It Works: The Uniform Distribution of Noise

The core intuition behind this approach lies in how label-blind pre-training alters the geometry of the feature space. In standard supervised learning, systematic label noise can cause a model to group completely unrelated images together simply because they share an incorrect label, creating polluted clusters and spurious shortcuts.

Self-supervised objectives (such as contrastive learning or masked autoencoding) completely ignore human-provided or heuristic labels. Instead, they optimize for data-centric properties, such as ensuring that two augmented views of the same image map to nearby points in the feature space.

As a result, the learned feature space organizes data into natural semantic clusters based purely on shared visual or structural attributes. What happens to the incorrect labels in this space? Pengcheng Xu et al. demonstrate in Uniform Noise Distribution and Compact Clusters: Unveiling the Success of Self-Supervised Learning in Label Noise that because the feature extraction process is entirely agnostic to labels, systematic label errors cannot distort the underlying representations. Instead, the label noise becomes uniformly distributed across the feature space. This uniform distribution transforms what would have been dangerous, systematic biases into manageable background interference, preventing the model from over-indexing on corrupted annotations.

This property allows models to extract high-quality, robust representations even from highly unstructured, web-scraped, or messy datasets, as shown by Florian Hinterwimmer et al. in From Self-supervised Learning to Transfer Learning with Musculoskeletal Radiographs and Paul Albert et al. in Addressing out-of-distribution label noise in webly-labelled data. Furthermore, because the pre-training phase does not involve class labels, the resulting features are naturally less prone to picking up label-based shortcuts or environmental biases, a mechanism leveraged by Nayeong Kim et al. in Learning Debiased Classifier with Biased Committee and Giacomo Capitani et al. in ClusterFix: A Cluster-Based Debiasing Approach without Protected-Group Supervision.

The Geometric Anchor for Noise Detection

Once a clean, self-supervised feature space is established, it can serve as a geometric "ground truth" reference to identify and correct corrupted labels. Because clean data points naturally cluster together based on semantic similarity, any sample whose assigned label contradicts the consensus of its geometric neighbors can be flagged as likely noise.

Supervised Feature Space (Polluted)       Self-Supervised Feature Space (Clean Anchor)
       [Cat]      [Dog*]                         [Cat] [Cat]     [Dog] [Dog]
          \        /                                \   /           \   /
        (Warped Cluster)                          (Cat Cluster)   (Dog Cluster)
                                                        |               |
                                                      [Dog*] <--- Flagged as Noise!

This structural consistency forms the basis for automated data cleaning and verification frameworks:

Regularization and Robust Downstream Adaptation

While pre-training provides a clean starting point, a key challenge remains: how do we fine-tune a model on a noisy dataset without letting the classifier eventually succumb to memorizing the noise?

One highly effective strategy is to use the self-supervised feature space as a rigid structural anchor during downstream training. In Mitigating Memorization of Noisy Labels via Regularization between Representations, Hao Cheng et al. introduce a representation regularizer that forces the supervised model to respect the distances established during self-supervised pre-training. This constraint prevents the classifier from drawing overly complex, unnatural decision boundaries just to fit corrupted labels.

Other frameworks maintain this resilience through collaborative training, distillation, or explicit architectural isolation:

By integrating self-supervised feedback loops directly into the training pipeline, models can continuously cross-reference incoming supervision signals with their internal structural consistency. This mechanism has been successfully applied to text segmentation by Mieradilijiang Maimaiti et al. in Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision and to continual learning by Myeongho Jeon et al. in Learning without Prejudices: Continual Unbiased Learning via Benign and Malignant Forgetting. Ultimately, anchoring models to the intrinsic geometry of data ensures high performance and structural integrity, even when the downstream supervision is sparse, ambiguous, or heavily corrupted (Yuval Onn).

Key Foundational Work

A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels (2020) — contributed 2 concept elements to this direction.
Self-Supervised Deep Multi-View Subspace Clustering (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_self_supervised_label_noise_resilience_d1e56963,
  title = {self-supervised label noise resilience},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-supervised-label-noise-resilience/rdr_d1e5696332c94b1bba847ae77635e988},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_self_supervised_label_noise_resilience_d1e56963_v1,
  title = {self-supervised label noise resilience},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-supervised-label-noise-resilience/rdr_d1e5696332c94b1bba847ae77635e988/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake