Dual Network Co Teaching for Noisy Labels

16 concepts16 papers1 proposal

As deep learning scales to massive datasets scraped from the web or generated via weak supervision, the assumption that every training label is correct completely breaks down. Training a standard deep neural network on this "dirty" data introduces a severe problem: while networks initially learn clean, regular patterns, they eventually memorize the incorrect labels, causing generalization performance to plummet.

To build trustworthy machine learning systems under such imperfect data, researchers have developed a powerful paradigm known as co-teaching or dual-network collaborative filtering. Instead of relying on a single model to clean its own data, this approach uses two distinct neural networks that interactively audit and filter training samples for one another.

The Self-Training Trap and the Memorization Effect

A naive approach to handling label noise is self-training: have a single network calculate the loss for each sample in a mini-batch, assume that samples with the lowest loss are correctly labeled, and update the network only on those "clean" samples.

This strategy fails because of confirmation bias. If a single network misclassifies a clean sample early in training due to its random initialization, it will perceive that sample as having a high loss and discard it. Conversely, it will continuously select samples that align with its own current errors, creating a self-reinforcing feedback loop. As Han et al. point out, this causes error accumulation, as the model quickly becomes blind to its own mistakes.

Co-teaching circumvents this trap by exploiting a fundamental property of deep networks: the memorization effect. Early in the training process, networks learn simple, clean underlying patterns before they begin to fit complex, random noise. This means that during the initial epochs, a sample's loss value is a highly reliable proxy for whether its label is correct.

How Co-Teaching Works: Peer Review for Data

The core mechanics of co-teaching are elegant. Rather than training one network, we initialize two identical architectures, Network A and Network B, with different random weights. Because their initializations differ, they will explore different optimization paths and develop distinct error surfaces.

[ Mini-Batch of Noisy Data ]
       /             \
      /               \
(Network A)       (Network B)
Evaluates Loss    Evaluates Loss
    |                 |
Selects Low-Loss  Selects Low-Loss
Samples (Clean)   Samples (Clean)
    \                 /
     \               /
   [ Cross-Update Step ]
Network A trains  Network B trains
  on B's choices    on A's choices

For every mini-batch of data, the training loop proceeds as follows:

  1. Parallel Evaluation: Both Network A and Network B ingest the same mini-batch and compute the loss for every sample.
  2. Small-Loss Selection: Each network independently identifies a subset of samples that yield the lowest loss. The size of this subset is controlled by a dynamic schedule that mirrors the estimated noise rate of the dataset, a property explored theoretically by Chen et al..
  3. The Cross-Update: Network A passes its selected low-loss samples to Network B, and Network B passes its selected samples to Network A. Each network updates its weights only on the samples chosen by its peer.

Why does this cross-filtering work? Because the networks are initialized differently, their memorization errors are conditionally independent. If Network A accidentally selects a noisy sample that happens to fit its current bias, Network B is unlikely to make that same specific mistake. By filtering the training signal through a peer network, the framework prevents individual model memorization and breaks the loop of error accumulation.

As surveyed by Sanchez et al., this collaborative filtering strategy forms the backbone of modern robust learning architectures.

Domain Adaptations and Variations

The dual-network co-teaching paradigm has proven remarkably versatile, adapting to the unique noise profiles of various domains:

Computer Vision and Object Detection

In object detection, discarding an entire image due to a single bad label is highly inefficient. Chadwick et al. adapted co-teaching to a "per-object" level, allowing parallel networks to filter out individual noisy bounding boxes while retaining the clean portions of an image. For cross-modal tasks like visible-infrared person re-identification, Yang et al. introduced frameworks to handle "Twin Noisy Labels"—where both individual identity annotations and cross-modal pairings are simultaneously corrupted.

Natural Language Processing

In text classification, particularly for low-resource languages where labels are crowdsourced, Zhu et al. demonstrated that dual-network cross-filtering effectively purges weak supervision noise. When analyzing large language models like BERT, Zhu et al. found that while pre-trained models are naturally resilient to random label noise, co-teaching is crucial for surviving complex, feature-dependent noise.

Medical Imaging and Healthcare

Medical datasets are frequently plagued by inter-observer variability and weak labels. In prostate cancer detection using micro-ultrasound, Gilany et al. combined dual-network sample exchange with evidential deep learning to provide both noise robustness and clinical confidence scores. Similarly, the authors of DPMIL utilized co-teaching within a multi-instance learning framework to filter out non-representative, noisy image patches when predicting breast cancer subtypes from pathological slides.

Cross-Domain and Unsupervised Learning

Co-teaching can also bridge structural divides. Wang et al. designed a collaborative system for electrical power time series where one network operates in the time domain and its peer operates in the frequency domain, allowing them to mutually correct domain-specific noise. Even in completely unsupervised tasks like stereo matching, Wang et al. showed that two networks can interactively exchange error-prone signals, such as occlusion maps, to prevent either model from locking onto geometric errors.

By replacing self-absorbed optimization with peer-driven validation, co-teaching ensures that deep models can extract clean knowledge from fundamentally chaotic data.

Key Foundational Work

Training Object Detectors With Noisy Data (2019) — contributed 1 concept elements to this direction.
Coupled-View Deep Classifier Learning from Multiple Noisy Annotators (2020) — contributed 1 concept elements to this direction.
Deep Learning Classification with Noisy Labels (2020) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 15 papers
  2. Version 1 · 2026-10-04 · 23 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_co_teaching_e55a9514,
  title = {co-teaching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/co-teaching/rdr_e55a9514ab254152af4dc330330697ed},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_co_teaching_e55a9514_v2,
  title = {co-teaching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/co-teaching/rdr_e55a9514ab254152af4dc330330697ed/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake