Recycling Noisy Samples for Semi-Supervised Learning

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2141 papers171 conceptsfrozen 2026-10-04

In the standard supervised learning paradigm, we assume our labels are ground truth. When they aren't—a common occurrence in web-crawled datasets or medical imaging—the traditional defense is to identify "noisy" samples and discard them. This is often done using the "small loss trick," where samples with high training loss are assumed to be mislabeled and are simply deleted from the training set.

However, a growing body of research suggests that discarding these samples is a mistake. Even if a label is wrong, the underlying data (the image pixels, the sentence structure, the satellite telemetry) still contains valuable information about the data distribution. Instead of throwing this information away, researchers are increasingly treating noisy samples as unlabeled data, integrating them back into the training process through semi-supervised learning (SSL).

The Intuition: Features Outlive Labels

The core insight behind this shift is that a sample's feature representation is often independent of its label's correctness. An image of a cat labeled "dog" is still a high-quality representation of what a cat looks like. If we discard it, we lose a data point that could have helped the model learn the visual geometry of cats.

By reclassifying noisy samples as unlabeled, we can apply SSL techniques like consistency regularization or pseudo-labeling. As Qi Wei and colleagues demonstrate in Self-Filtering: A Noise-Aware Sample Selection for Label Noise with Confidence Penalization, this allows a model to leverage the structural information of noisy samples without being misled by their incorrect labels. This "recycling" prevents the loss of difficult "boundary" samples that are often the first to be flagged as noise but are crucial for high-performance classification.

Mechanisms of Data Recovery

How exactly do we "use" a sample once we've decided its label is untrustworthy? The research points to three primary mechanisms:

1. Consistency Regularization

If we treat a noisy sample as unlabeled, we can force the model to produce consistent predictions for different augmented versions of that sample. In FedNoiL: A Simple Two-Level Sampling Method for Federated Learning with Noisy Labels, Tianyi Zhou and co-authors show that this strategy allows models to learn from underlying features even in high-noise environments like federated learning. Similarly, Zhaowei Zhu et al. use KL-divergence in CORES$^2$ to ensure that the model’s internal representation of a noisy image remains stable, effectively using the noise as a source of unsupervised regularization.

2. Pseudo-Labeling and Rectification

Rather than accepting a noisy label or ignoring it, we can let the model "guess" a better one. Yanyan Wang et al. describe this in CRLN, where they use a small set of clean data to "rectify" noisy pseudo-labels in 3D medical segmentation. This turns a potentially harmful data point into a useful one by aligning it with learned prototypes.

3. Latent Semantic Mining

In cross-modal tasks, "noise" often means a mismatch between an image and its caption. Haochen Han et al. argue in Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval that these pairs shouldn't be discarded. Instead, they use Optimal Transport to "rematch" the image with a more appropriate caption from the same batch, turning noise into a rich source of supervision.

When Semi-Supervised Recycling Works Best

The effectiveness of treating noise as unlabeled data isn't universal; it depends on the causal structure of the data. Yu Yao and Bo Han explore this in Which is Better for Learning with Noisy Labels: The Semi-supervised Method or Modeling Label Noise?. They find that SSL-based recycling is particularly powerful in "anticausal" settings—where the label causes the features (like image recognition, where the concept of "dog" determines the pixels). In these cases, the feature distribution is deeply entangled with the class boundaries, making the unlabeled features highly informative for recovering the true labels.

Beyond Simple Filtering

The transition from "filtering" to "recycling" represents a broader shift in the field toward data efficiency. In Exploiting Noisy Data in Distant Supervision Relation Classification, Kaijia Yang and colleagues demonstrate that reclaiming noisy sentences as unlabeled data in a Variational Autoencoder (VAE) significantly improves linguistic pattern recognition.

By moving away from hard-filtering—which Ruotian Ma et al. note in SENT can lead to error propagation and permanent loss of information—researchers are building models that are not just robust to noise, but capable of thriving on it. The "garbage" of yesterday's datasets is becoming the "unlabeled gold" of today's semi-supervised frameworks.

Papers in this version

Showing 60 of 141 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 132 papers
  2. Version 1 · 2026-10-04 · 141 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_semi_supervised_noise_mitigation_13b4c7f8,
  title = {semi-supervised noise mitigation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semi-supervised-noise-mitigation/rdr_13b4c7f81b824010aebb32f597fdb26e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_recycling_noisy_samples_for_semi_supervi_13b4c7f8_v1,
  title = {Recycling Noisy Samples for Semi-Supervised Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recycling-noisy-samples-for-semi-supervised-learning/rdr_13b4c7f81b824010aebb32f597fdb26e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}