Collaborative Peer Supervision for Robust Learning with Noisy Labels

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 223 papers25 conceptsfrozen 2026-10-04

Deep learning models are notoriously "hungry" for high-quality data, but real-world datasets—from medical records to web-scraped images—are often riddled with incorrect or "noisy" labels. When a standard neural network encounters these errors, it initially learns the correct patterns but eventually begins to "memorize" the noise, leading to poor generalization.

A powerful family of techniques, often referred to as Co-teaching or Peer Learning, addresses this by training two or more networks simultaneously. Instead of trusting their own potentially biased predictions, these networks act as "peers" that filter the training data for one another.

The Small-Loss Intuition

The foundation of this research direction is the observation that neural networks learn clean, simple patterns before they learn complex noise. In the early stages of training, samples that result in a low loss (where the model's prediction is close to the provided label) are statistically much more likely to be correctly labeled than those with a high loss.

However, if a single model simply selects its own low-loss samples to train on, it creates a "self-confirmation bias." It will only learn what it already knows, reinforcing its own initial errors and ignoring difficult but correct examples.

Breaking the Feedback Loop with Peer Supervision

To break this cycle, researchers like Jiazhan Feng and Dawei Zhu propose using two independent networks. In a typical Co-teaching setup:

  1. Network A and Network B both view a mini-batch of data.
  2. Each network identifies which samples in that batch have the lowest loss according to its current parameters.
  3. Network A passes its "clean" samples to Network B to learn from, and vice versa.

Because the two networks are initialized differently or have different architectures, they make different mistakes. By exchanging "clean" samples, they provide a second opinion that prevents either model from overfitting to the specific noise patterns it might have otherwise memorized.

Diversifying the Peer Architecture

While early versions used identical architectures, recent work has shown that increasing the "diversity" between peers improves robustness. Qinming Zhang introduced the Cross-denoising Network (CD-Net), which uses two different neural architectures with distinct inductive biases. This ensures that the models don't converge on the same errors, making the mutual filtering process more effective even when up to 90% of labels are faulty.

Similarly, Shikun Li developed a Coupled-View Learning (CVL) framework that treats raw data and noisy labels as separate "views," using mutual supervision to handle the specific challenges of crowdsourced data where multiple annotators provide conflicting labels.

Applications Across Domains

This collaborative filtering mechanism has proven remarkably versatile:

  • Medical Imaging: In high-stakes environments like prostate cancer detection, Mahdi Gilany and Amoon Jamzad use co-teaching to handle "messy" clinical labels, combining it with evidential deep learning to provide confidence scores. Hong Liu applied a similar strategy to predict breast cancer subtypes from pathology images, outperforming senior pathologists by filtering out non-informative tissue patches.
  • Natural Language Processing: Dawei Zhu demonstrated that while models like BERT are somewhat resilient to random noise, they benefit significantly from co-teaching when facing the complex, feature-dependent noise found in low-resource African languages.
  • Multi-modal Retrieval: For tasks like matching images to text descriptions, Shuo Yang introduced BiCro, which uses dual networks to rectify "noisy correspondences" in web-scraped data by ensuring similarity consistency across modalities.

Beyond Simple Filtering

The latest iterations of this idea move beyond just discarding noisy samples. The Contrastive Co-Transformer (CCT) by Han Yan combines peer learning with contrastive loss. Instead of ignoring high-loss data, the system uses it in an unsupervised way, learning the underlying structure of the data even when the labels cannot be trusted.

By shifting from "self-learning" to "peer-teaching," these frameworks allow AI to find the signal within the noise, making deep learning viable in domains where perfect data is an impossible luxury.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 15 papers
  2. Version 1 · 2026-10-04 · 23 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_co_teaching_e55a9514,
  title = {co-teaching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/co-teaching/rdr_e55a9514ab254152af4dc330330697ed},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_collaborative_peer_supervision_for_robus_e55a9514_v1,
  title = {Collaborative Peer Supervision for Robust Learning with Noisy Labels},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/collaborative-peer-supervision-for-robust-learning-with-noisy-labels/rdr_e55a9514ab254152af4dc330330697ed/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}