Contrastive Credibility Propagation for Reliable Semi-supervised Learning

Brody Kutt, Pralay Ramteke, Xavier Mignot, Pamela Toman, Nandini Ramanan, Sujit Rokka Chhetri, Shan Huang, Min Du, William HewlettAAAI 2024 · DOI

TL;DR

This paper introduces Contrastive Credibility Propagation (CCP), a novel semi-supervised learning (SSL) algorithm designed to be robust across various real-world data imperfections like few labels, noisy labels, and imbalanced class distributions. CCP achieves this by iteratively refining pseudo-labels using a "credibility vector" that quantifies uncertainty, allowing it to consistently outperform supervised baselines even when data quality is unknown.

What Problem Does This Paper Address?

Semi-supervised learning (SSL) aims to leverage unlabeled data to improve model performance beyond what's achievable with labeled data alone. However, many existing SSL methods perform poorly when faced with real-world data challenges, such as having very few labeled examples per class, encountering out-of-distribution samples in the unlabeled set, dealing with incorrect labels, or facing imbalanced class distributions between labeled and unlabeled data. A key issue is that these methods often assume clean, balanced datasets and can even be negatively impacted by their own generated "pseudo-labels" if they are inaccurate, leading to confirmation bias. This paper tackles the critical need for an SSL algorithm that is reliable and performs well across a spectrum of these common data imperfections, without requiring prior knowledge of which specific issues are present.

Methods

The core of CCP is an iterative process of pseudo-label refinement and classifier training.

  • Credibility Vectors: Instead of using hard pseudo-labels or standard softmax outputs, CCP introduces "credibility vectors." These vectors represent a sample's confidence in belonging to each class, where a score of 1 means maximum confidence, 0 means uncertainty, and -1 means maximum confidence in not belonging to a class. This richer representation helps capture ambiguity.
  • Contrastive Credibility Propagation (CCP) Iteration: The algorithm iteratively refines these credibility vectors for unlabeled data. This involves:
  • Soft Contrastive Loss (Lssc): A generalized contrastive loss that uses credibility vectors to define "soft" positive pairs, allowing for uncertain relationships rather than discrete assignments. This helps propagate information and learn representations.
  • Credibility Adjustments: After generating new credibility vectors, they are averaged across iterations, scaled, and clipped to a [0, 1] range. This process helps to stabilize and refine the pseudo-labels.
  • DKL-Based Subsampling: To prevent instability caused by resetting too many weak pseudo-labels, a Kullback-Leibler (KL) divergence-based subsampling strategy is employed to decide which pseudo-labels to reset, ensuring the overall class distribution remains consistent.
  • Classifier Training: Once the pseudo-labels are refined through CCP iterations, a final classifier is trained using these soft labels.

Datasets

The experiments were conducted on CIFAR-10 and CIFAR-100 datasets. The authors created new experimental setups to simulate five specific data variables: few-label, open-set, noisy-label, and class distribution imbalance/misalignment, at varying levels of severity.

Key Insights and Ideas

  • Credibility is Key: The paper argues that sensitivity to pseudo-label errors is the root cause of SSL failures. By using "credibility vectors" to represent uncertainty more nuancedly than traditional softmax outputs, CCP can better handle ambiguous or incorrect pseudo-labels.
  • Robustness over Specificity: Instead of optimizing for robustness to a single data variable, CCP aims to be robust to all common data imperfections simultaneously, making it practical for real-world scenarios where data quality is often unknown.
  • Unifying SSL and Noisy Label Learning: CCP effectively combines techniques for semi-supervised learning with methods for handling noisy labels, a significant step towards a more general-purpose SSL solution.
  • Iterative Refinement Matters: The iterative nature of CCP, where pseudo-labels are continuously refined and propagated, is crucial for its ability to overcome initial errors and converge to reliable predictions.

Key Results

  • Consistent Superiority: CCP consistently outperforms a supervised baseline across all five tested data variable scenarios (few-label, open-set, noisy-label, class distribution imbalance/misalignment) at various severity levels on both CIFAR-10 and CIFAR-100.
  • Robustness to Noise: CCP shows a significant advantage in the noisy-label scenario, where other algorithms often fail catastrophically.
  • Minimal Performance Drop: On CIFAR-10, CCP's accuracy never drops below 90% even with data imperfections, representing only a 5% drop from its base-case performance.
  • DKL-Based Subsampling Benefit: The DKL-based subsampling strategy was found to provide substantial benefits to pseudo-label accuracy, especially at higher data variable severity.

Limitations and Future Work

  • Computational Expense: CCP iterations can be computationally taxing, especially with a large number of epochs (Ξ). The authors mitigate this by using a pretrained encoder and early stopping.
  • Tuning dmax: While the DKL-based subsampling strategy is designed to generalize, instability can occur if the divergence limit (dmax) is too large, requiring tuning.
  • Future Work: The authors suggest augmenting CCP with components from other successful SSL methods, such as consistency training with weak/strong augmentations and instance/semantic similarity, to further improve label efficiency while retaining CCP's reliability.

Summary updated

BibTeX
@inproceedings{kutt2024contrastive,
  author = {Brody Kutt and Pralay Ramteke and Xavier Mignot and Pamela Toman and Nandini Ramanan and Sujit Rokka Chhetri and Shan Huang and Min Du and William Hewlett},
  title = {{Contrastive Credibility Propagation for Reliable Semi-supervised Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i19.30124},
}
Figure