Mitigating Confirmation Bias with Collaborative Dual Network Training

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2150 papers200 conceptsfrozen 2026-10-04

In semi-supervised and noisy-label learning, models often have to act as their own teachers. This "self-training" approach is intuitive: a model makes a prediction on unlabeled data, treats that prediction as a "pseudo-label," and then updates its weights to match it. However, this creates a dangerous feedback loop known as confirmation bias. If a model is even slightly wrong about a sample, it will reinforce that error during training, leading to a "self-fulfilling prophecy" where the model becomes increasingly confident in its own mistakes.

To break this cycle, researchers have converged on a "buddy system" architecture: using two or more networks to supervise each other.

The Self-Training Trap

The fundamental problem with single-model self-training is the lack of an external objective. As noted by Yinghao Xu and colleagues in their work on Cross-Model Pseudo-Labeling, a single model cannot easily distinguish between a correct prediction and a confident error. Once an error is incorporated into the training set as a pseudo-label, the model's gradient descent will actively work to "memorize" that noise.

Hao Cheng and Mingcai Chen describe this as a loop where "two wrongs don't make a right." In their Robust Label Refurbishment framework, they argue that simply discarding suspicious data leads to information loss, while keeping it leads to confirmation bias. The solution is not to train harder, but to train collaboratively.

The Peer Supervision Mechanism

The core mechanism of this research direction is cross-supervision. Instead of one model talking to itself, two independent networks (often called "peer networks") are trained simultaneously.

  1. Model A predicts a label for an unlabeled sample.
  2. Model B uses that prediction as its ground truth for that sample.
  3. Model B predicts a label for a different sample (or a different view of the same sample).
  4. Model A uses Model B's prediction as its ground truth.

By decoupling the source of the label from the model being updated, the system ensures that a model does not exclusively learn from its own potentially erroneous predictions. As Kun Zhan and Chaoxi Niu demonstrate in Mutual Teaching for Graph Convolutional Networks, this exchange allows models to filter out noise and explore the data manifold more effectively than a lone model could.

Diversity as a Denoising Filter

For peer supervision to work, the two models must not be identical clones. If they make the exact same mistakes, the confirmation bias remains. Researchers use several strategies to ensure "disagreement" or diversity between the peers:

  • Architectural Diversity: Xinghua Zhang and Bowen Yu show that using models with different inductive biases—such as a Tagger and a Classifier—prevents them from overfitting to the same noise patterns in distantly-supervised NER.
  • View Diversity: In LiDAR segmentation, Yuyuan Liu and Gustavo Carneiro use ItTakesTwo to enforce consistency between "Range" and "Voxel" views of the same data, ensuring pseudo-labels are validated across different structural representations.
  • Bias Diversity: Jaemin Na and Wonjun Hwang propose FixBi, which trains one model to be "source-dominant" and another to be "target-dominant." This allows the models to bridge domain gaps by teaching each other from opposing perspectives.

Robustness Through Consensus

A recurring pattern in these papers is the use of confidence-based selection. In CroSel, Hongxin Wei and Lei Feng use the "small-loss" property: neural networks tend to learn clean, simple patterns before they begin to memorize complex noise. By having peers identify "high-confidence" or "low-loss" samples for each other, the networks can effectively filter out noisy labels before they ever enter the training loop.

This collaborative filtering is even effective in extreme scenarios. For instance, Gengyu Lyu and Bohang Sun apply this to Partial-Label Learning, where each image has a set of candidate labels. Their dual-network approach identifies likely false negatives for the peer, preventing the model from reinforcing the ambiguity present in the original annotations.

Broad Applications

While the mechanism remains consistent, the applications are diverse:

By shifting from "self-teaching" to "peer-teaching," these methods transform the liability of model bias into a collaborative asset, allowing deep learning systems to find the signal in increasingly noisy and unlabeled data.

Papers in this version

Showing 60 of 150 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 162 papers
  2. Version 1 · 2026-10-04 · 150 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_collaborative_pseudo_labeling_33112b95,
  title = {collaborative pseudo-labeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/collaborative-pseudo-labeling/rdr_33112b95f6db48bbafc9124408c2bd5e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_mitigating_confirmation_bias_with_collab_33112b95_v1,
  title = {Mitigating Confirmation Bias with Collaborative Dual Network Training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-confirmation-bias-with-collaborative-dual-network-training/rdr_33112b95f6db48bbafc9124408c2bd5e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}