TL;DR
This paper introduces EPASS, a simple yet effective method to improve semi-supervised learning (SSL) by using an ensemble of multiple "projectors" to create more robust and less biased embeddings. EPASS significantly boosts the performance of existing contrastive joint-training SSL frameworks like CoMatch and SimMatch, achieving state-of-the-art results on large-scale datasets like ImageNet with minimal computational overhead.
What Problem Does This Paper Address?
Semi-supervised learning (SSL) has made remarkable progress by leveraging large amounts of unlabeled data alongside a small set of labeled data. However, many state-of-the-art SSL methods, particularly those based on contrastive learning, tend to become increasingly complex. A critical issue they face is "confirmation bias," where the model can become overly reliant on its own incorrect predictions (pseudo-labels), leading to a degradation of learned representations and overall performance. This is particularly problematic in joint-training schemes where the easy-to-learn representations can dominate, creating biased embeddings that are then fed back into the learning process, creating a vicious cycle. Existing methods often struggle to mitigate this bias effectively.
Methods
The core innovation of this paper is the Ensemble Projectors Aided for Semi-supervised Learning (EPASS). Instead of using a single projector (a small neural network, typically an MLP) to transform the encoder's output into low-dimensional embeddings, EPASS utilizes an ensemble of multiple projectors. These projectors are randomly initialized, allowing them to learn diverse feature representations from different perspectives. The embeddings from these multiple projectors are then averaged to create a more generalized and less biased representation.
This ensemble approach is designed as a "plug-and-play" module that can be easily integrated into existing contrastive joint-training SSL frameworks like CoMatch and SimMatch. The authors hypothesize that this ensemble strategy helps to:
- Mitigate Overfitting: By averaging diverse representations, the model is less likely to overfit to the specific characteristics of a single projector.
- Reduce Confirmation Bias: The ensemble of projectors provides a more robust and less biased embedding space, which in turn improves the accuracy of pseudo-labels and the overall EMA teacher.
- Enhance Generalization: The diverse views from multiple projectors lead to more distinguishable and informative representations.
The paper also details how EPASS is applied to SimMatch and CoMatch, modifying their instance similarity calculations and contrastive loss formulations to incorporate the averaged ensemble embeddings.
Datasets
The authors evaluate EPASS on several standard benchmarks for semi-supervised learning:
- CIFAR-10/100: Widely used image classification datasets.
- SVHN (Street View House Numbers): A dataset of house numbers.
- STL-10: Another image classification dataset.
- ImageNet (ILSVRC-2012): A large-scale, challenging image classification dataset.
They also conduct experiments on imbalanced versions of CIFAR-10 and CIFAR-100 (CIFAR-10-LT, CIFAR-100-LT) to assess performance in scenarios with skewed class distributions. Additionally, they evaluate EPASS within the USB framework using pre-trained Vision Transformers (ViT).
Key Insights and Ideas
- The Power of Ensemble for Embeddings: The paper demonstrates that simply averaging embeddings from multiple randomly initialized projectors can significantly improve SSL performance. This is a surprisingly simple yet effective way to combat confirmation bias.
- Projectors as a Source of Diversity: Unlike data augmentation which creates different views of the data, EPASS creates different views of the representation through multiple projectors. This offers a complementary approach to improving robustness.
- Minimal Overhead, Maximum Impact: EPASS is presented as a lightweight module that adds minimal computational cost while yielding substantial performance gains, making it highly practical.
- Debiasing is Key: The core idea is that by debiasing the learned embeddings, the entire SSL pipeline, including pseudo-label generation and the EMA teacher, becomes more reliable.
Key Results
EPASS consistently improves the performance of contrastive joint-training SSL methods:
- ImageNet Performance: EPASS significantly boosts SimMatch and CoMatch on ImageNet. For example, with 10% labeled data, SimMatch + EPASS achieves a top-1 error rate of 24.70% (compared to 41.15% for SimMatch alone), and CoMatch + EPASS achieves 25.90% (compared to 42.17% for CoMatch alone). These results are state-of-the-art for these methods.
- CIFAR-10/100, SVHN, STL-10: EPASS also shows improvements on these smaller datasets, achieving new state-of-the-art results on SVHN and STL-10.
- Imbalanced SSL: EPASS demonstrates strong performance on imbalanced datasets, outperforming other methods and improving SimMatch and CoMatch significantly.
- Calibration: Reliability diagrams and Expected Calibration Error (ECE) show that EPASS reduces overconfidence and improves model calibration, especially on datasets with limited labels.
- Convergence Speed: Figure 3 shows that models with EPASS converge faster and reach higher accuracies compared to their counterparts without EPASS.
Limitations and Future Work
- Generalization to Other Domains: The authors acknowledge that while EPASS is effective for image classification, its applicability to other domains like natural language processing or speech recognition is an open question. Future work could explore adapting EPASS to these areas.
- Beyond Classification: The paper focuses on classification tasks. Future research could investigate EPASS's utility in more complex computer vision tasks such as object detection or segmentation.
- Ensemble Strategies: While averaging embeddings was found to be effective, the paper briefly mentions exploring other ensemble strategies like concatenation and summation. Further investigation into optimal ensemble methods could be beneficial.
Summary updated