Abstract
Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model's predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -- just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch's success.
Introduction
Deep learning typically requires massive labeled datasets, which are often expensive or difficult to obtain. Semi-supervised learning (SSL) addresses this by utilizing a small amount of labeled data alongside a much larger pool of unlabeled data. Recent trends in SSL have favored complex "holistic" methods like MixMatch and ReMixMatch, which combine multiple techniques but introduce significant implementation complexity and numerous hyperparameters. FixMatch seeks to simplify this pipeline, demonstrating that a streamlined approach can achieve state-of-the-art results by combining consistency regularization and pseudo-labeling.
Methods: The Weak-to-Strong Framework
FixMatch is built on a "weak-to-strong" consistency framework. For every unlabeled image in a training batch, the algorithm follows a straightforward process:
- Weak Augmentation: A simple transformation (e.g., horizontal flipping or shifting) is applied. The model predicts a class distribution for this version.
- Pseudo-labeling & Thresholding: If the model's confidence for the top class exceeds a threshold $\tau$ (typically 0.95), the prediction is converted into a "hard" one-hot pseudo-label.
- Strong Augmentation: The same original image is heavily distorted using techniques like RandAugment or CTAugment, followed by Cutout.
- Consistency Loss: The model is trained to predict the previously generated pseudo-label when fed the strongly-augmented version of the image, using standard cross-entropy loss.
The final loss is simply the sum of the standard supervised loss on labeled data and this unsupervised consistency loss. Unlike previous methods, FixMatch does not require complex "sharpening" of distributions or annealing the weight of the unlabeled loss over time.
Experimental Results
The authors evaluated FixMatch on several standard computer vision benchmarks, including CIFAR-10/100, SVHN, STL-10, and ImageNet. FixMatch consistently outperformed or matched more complex state-of-the-art methods, particularly in extremely label-scarce regimes.
Key performance highlights include:
- CIFAR-10: Achieved 94.93% accuracy with 250 labels and 88.61% with just 40 labels (4 per class).
- ImageNet: On the 10% labeled split, FixMatch achieved a top-1 error rate of 28.54%, significantly better than previous SOTA methods like UDA.
- Extreme Low-Label: On CIFAR-10 with a single label per class, the model reached a median accuracy of 64.3%, and as high as 78% if the labeled images were carefully chosen.
Key Insights
- Weak-to-Strong Synergy: The core innovation is that artificial labels should be easy to generate (weak augmentation) but challenging to predict (strong augmentation). This forces the model to learn robust, invariant features.
- Implicit Curriculum Learning: The high confidence threshold $\tau$ acts as a natural curriculum. Early in training, the model ignores most unlabeled data when its predictions are uncertain. As the model improves, more images pass the threshold, providing increasingly difficult training signals.
- Simplicity as a Feature: With fewer hyperparameters, FixMatch is easier to analyze and implement. Ablation studies show that standard training choices, like the choice of optimizer and the learning rate schedule, are just as critical as the SSL algorithm itself.
Limitations and Future Work
While highly effective, FixMatch shows a slight performance gap compared to ReMixMatch on CIFAR-100, though this can be mitigated by adding "Distribution Alignment" at the cost of increased complexity. Additionally, performance in extreme low-label regimes is highly sensitive to the specific images chosen for labeling; "prototypical" images yield much better results than outliers. Finally, the strong augmentations used are currently image-specific; adapting FixMatch to other domains like NLP or audio remains an area for future research.
Summary updated