TL;DR
This paper introduces ReLaB, a novel unsupervised method to significantly boost semi-supervised learning performance in extremely low-label scenarios. By leveraging self-supervised features for label propagation and then carefully selecting reliable pseudo-labels, ReLaB effectively expands the training set, leading to dramatic error rate reductions.
What Problem Does This Paper Address?
Training deep convolutional neural networks (CNNs) for image classification typically requires vast amounts of labeled data. While unlabeled data is abundant, the annotation process is a major bottleneck due to its cost and time intensity. Semi-supervised learning (SSL) aims to mitigate this by using a small set of labeled data alongside a large amount of unlabeled data. However, existing SSL methods still struggle when the labeled data is extremely scarce (e.g., just one labeled example per class). This paper tackles this challenge by proposing a pre-processing step that intelligently generates more reliable labels from unlabeled data, thereby enhancing the effectiveness of downstream SSL algorithms.
Methods
ReLaB is a four-step unsupervised pre-processing pipeline:
- Self-Supervised Feature Learning: The process begins by learning meaningful, discriminative latent features from all available data (both labeled and unlabeled) using self-supervised learning techniques. This step is crucial for capturing the underlying structure of the data without relying on human-provided labels.
- Label Propagation: Using the learned self-supervised features, a label propagation algorithm (based on graph diffusion) is employed to spread the initial few labels to the entire unlabeled dataset. This effectively creates a fully "labeled" dataset, but with a significant amount of noisy, incorrect labels.
- Reliable Sample Selection: To combat the noise introduced in the previous step, a label noise detection algorithm is used. ReLaB identifies a subset of "reliable" samples from the propagated labels, aiming to create a cleaner, more trustworthy extended labeled set. This is achieved by selecting samples with low cross-entropy loss, a common trick in label noise literature, but adapted here to handle the specific challenges of label propagation noise.
- Semi-Supervised Training: Finally, a standard semi-supervised learning algorithm is trained on this newly created, reliable extended labeled dataset. The idea is that by providing a larger, cleaner set of labels, the SSL algorithm can achieve much better performance, especially in extreme low-label regimes.
The authors emphasize that the choice of self-supervised method and network architecture significantly impacts the quality of the propagated labels.
Datasets
The proposed ReLaB method was evaluated on the following standard image classification datasets:
- CIFAR-10: 10 classes, 32x32 images.
- CIFAR-100: 100 classes, 32x32 images.
- mini-ImageNet: 100 classes, 84x84 images.
Key Insights and Ideas
- Self-supervised features are critical for label propagation: The quality of the features learned via self-supervision directly dictates how well labels can be propagated. Poor features lead to high noise in the propagated labels.
- Label propagation inherently introduces noise: While effective for expanding labels, the process is not perfect and generates many incorrect labels, especially when dealing with visually similar classes or subtle differences.
- A robust label selection mechanism is essential: Simply using all propagated labels would harm performance. ReLaB's reliable sample selection step is key to filtering out noisy labels and creating a usable extended dataset.
- Class-balanced noise is a challenge: The noise introduced by label propagation is not uniformly distributed across classes, making simple noise handling techniques less effective. ReLaB's approach aims to address this.
- ReLaB acts as a powerful pre-processor: It's not a standalone SSL method but a way to enhance existing SSL algorithms, making them more effective in data-scarce scenarios.
Key Results
- Dramatic error rate reduction: ReLaB significantly improves SSL performance. For example, on CIFAR-10 with just 1 labeled sample per class, ReLaB + ReMixMatch achieved an error rate of 8.46% when the sample was representative, compared to 50.62% for ReMixMatch alone.
- Scalability: The method shows promise on higher resolution images like mini-ImageNet.
- Importance of self-supervised method and architecture: Experiments showed that using iMix with a ResNet-50 architecture yielded the lowest label noise after propagation.
- Robustness to label noise: ReLaB combined with SSL methods outperformed supervised methods trained on noisy data.
Limitations and Future Work
- Sensitivity to representative samples: The performance can still be significantly impacted by the representativeness of the initial few labeled samples, especially in the "1 sample per class" scenario. The paper explores this by using different subsets of samples.
- Computational cost: The self-supervised pre-training and label propagation steps add computational overhead.
- Future work could explore: Applying ReLaB to other domains like medical imaging where expert annotators are required and data is scarce. Further investigating more robust label noise handling techniques tailored for propagation-induced noise.
Summary updated