TL;DR
This paper introduces Contrastive Credibility Propagation (CCP), a novel semi-supervised learning (SSL) algorithm designed to be robust across various real-world data imperfections like few labels, noisy labels, and imbalanced class distributions. CCP achieves this by iteratively refining pseudo-labels using a "credibility vector" that quantifies uncertainty, allowing it to consistently outperform supervised baselines even when data quality is unknown.
What Problem Does This Paper Address?
Semi-supervised learning (SSL) aims to leverage unlabeled data to improve model performance beyond what's achievable with labeled data alone. However, many existing SSL methods perform poorly when faced with real-world data challenges, such as having very few labeled examples per class, encountering out-of-distribution samples in the unlabeled set, dealing with incorrect labels, or facing imbalanced class distributions between labeled and unlabeled data. A key issue is that these methods often assume clean, balanced datasets and can even be negatively impacted by their own generated "pseudo-labels" if they are inaccurate, leading to confirmation bias. This paper tackles the critical need for an SSL algorithm that is reliable and performs well across a spectrum of these common data imperfections, without requiring prior knowledge of which specific issues are present.
Methods
The core of CCP is an iterative process of pseudo-label refinement and classifier training.
- Credibility Vectors: Instead of using hard pseudo-labels or standard softmax outputs, CCP introduces "credibility vectors." These vectors represent a sample's confidence in belonging to each class, where a score of 1 means maximum confidence, 0 means uncertainty, and -1 means maximum confidence in not belonging to a class. This richer representation helps capture ambiguity.
- Contrastive Credibility Propagation (CCP) Iteration: The algorithm iteratively refines these credibility vectors for unlabeled data. This involves:
- Soft Contrastive Loss (Lssc): A generalized contrastive loss that uses credibility vectors to define "soft" positive pairs, allowing for uncertain relationships rather than discrete assignments. This helps propagate information and learn representations.
- Credibility Adjustments: After generating new credibility vectors, they are averaged across iterations, scaled, and clipped to a [0, 1] range. This process helps to stabilize and refine the pseudo-labels.
- DKL-Based Subsampling: To prevent instability caused by resetting too many weak pseudo-labels, a Kullback-Leibler (KL) divergence-based subsampling strategy is employed to decide which pseudo-labels to reset, ensuring the overall class distribution remains consistent.
- Classifier Training: Once the pseudo-labels are refined through CCP iterations, a final classifier is trained using these soft labels.
Datasets
The experiments were conducted on CIFAR-10 and CIFAR-100 datasets. The authors created new experimental setups to simulate five specific data variables: few-label, open-set, noisy-label, and class distribution imbalance/misalignment, at varying levels of severity.
Key Insights and Ideas
- Credibility is Key: The paper argues that sensitivity to pseudo-label errors is the root cause of SSL failures. By using "credibility vectors" to represent uncertainty more nuancedly than traditional softmax outputs, CCP can better handle ambiguous or incorrect pseudo-labels.
- Robustness over Specificity: Instead of optimizing for robustness to a single data variable, CCP aims to be robust to all common data imperfections simultaneously, making it practical for real-world scenarios where data quality is often unknown.
- Unifying SSL and Noisy Label Learning: CCP effectively combines techniques for semi-supervised learning with methods for handling noisy labels, a significant step towards a more general-purpose SSL solution.
- Iterative Refinement Matters: The iterative nature of CCP, where pseudo-labels are continuously refined and propagated, is crucial for its ability to overcome initial errors and converge to reliable predictions.
Key Results
- Consistent Superiority: CCP consistently outperforms a supervised baseline across all five tested data variable scenarios (few-label, open-set, noisy-label, class distribution imbalance/misalignment) at various severity levels on both CIFAR-10 and CIFAR-100.
- Robustness to Noise: CCP shows a significant advantage in the noisy-label scenario, where other algorithms often fail catastrophically.
- Minimal Performance Drop: On CIFAR-10, CCP's accuracy never drops below 90% even with data imperfections, representing only a 5% drop from its base-case performance.
- DKL-Based Subsampling Benefit: The DKL-based subsampling strategy was found to provide substantial benefits to pseudo-label accuracy, especially at higher data variable severity.
Limitations and Future Work
- Computational Expense: CCP iterations can be computationally taxing, especially with a large number of epochs (Ξ). The authors mitigate this by using a pretrained encoder and early stopping.
- Tuning dmax: While the DKL-based subsampling strategy is designed to generalize, instability can occur if the divergence limit (
dmax) is too large, requiring tuning. - Future Work: The authors suggest augmenting CCP with components from other successful SSL methods, such as consistency training with weak/strong augmentations and instance/semantic similarity, to further improve label efficiency while retaining CCP's reliability.
Summary updated