FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, Chun-Liang Li

TL;DR

FixMatch is a streamlined semi-supervised learning (SSL) algorithm that achieves state-of-the-art results by combining two classic ideas: consistency regularization and pseudo-labeling. By using high-confidence predictions on "weakly-augmented" images to supervise training on "strongly-augmented" versions of the same data, it reaches 94.93% accuracy on CIFAR-10 with only 25 labels per class.

What Problem Does This Paper Address?

Deep learning thrives on massive datasets, but labeling data is expensive, slow, and often requires domain expertise (e.g., a radiologist labeling X-rays). Semi-supervised learning (SSL) aims to solve this by training models on a tiny sliver of labeled data alongside a mountain of unlabeled data.

Before FixMatch, the field was trending toward increasingly complex "everything-but-the-kitchen-sink" methods. These algorithms (like MixMatch and ReMixMatch) involved complicated pipelines, multiple loss terms, and a dizzying array of hyperparameters. The authors of FixMatch asked: Can we achieve better results by simplifying the process?

Methods

FixMatch is built on a deceptively simple "weak-to-strong" consistency loop that leverages the model's own predictions to guide its learning on unlabeled data.

FixMatch Algorithm Diagram
  1. Weak Augmentation: For an unlabeled image, the model first sees a "weakly-augmented" version (simple horizontal flips or shifts).
  2. Pseudo-Labeling: The model generates a prediction for this image. If the model is highly confident (probability $\ge 0.95$), this prediction is converted into a "hard" one-hot label (a pseudo-label).
  3. Strong Augmentation: The model then sees a "strongly-augmented" version of the same image (using tools like RandAugment or CTAugment, which apply heavy distortions, color jitters, and "Cutout" holes).
  4. Consistency Loss: The model is trained to predict the pseudo-label when looking at the strongly-augmented image.

Essentially, the model uses its own "easy" guesses to teach itself how to handle "difficult" versions of the same data.

Datasets

The authors evaluated FixMatch on standard SSL benchmarks:

  • CIFAR-10 and CIFAR-100: Small-scale image classification.
  • SVHN (Street View House Numbers): Digit recognition.
  • STL-10: A dataset specifically designed for SSL with a large set of unlabeled images.
  • ImageNet: The gold standard for large-scale computer vision.

Notably, they pushed these benchmarks to the extreme, testing scenarios with as few as 1 to 4 labels per class.

Key Insights and Ideas

  • The "Free" Curriculum: FixMatch naturally creates a curriculum. Early in training, the model is unconfident, so the thresholding mechanism ignores most unlabeled data. As the model improves, it becomes more confident, "unlocking" more unlabeled data to train on.
  • The Power of the Gap: The core engine of FixMatch is the disparity between the weak and strong augmentations. The ablation studies showed that using strong augmentation to generate the pseudo-label causes the model to diverge, while using no augmentation leads to overfitting.
  • Simplicity is a Feature: By stripping away complex components like "distribution alignment" or "sharpening," the authors found that basic training choices—like using standard SGD with momentum instead of Adam and maintaining a strict weight decay—had a massive impact on performance.
  • One Label is (Sometimes) All You Need: The paper demonstrates that with just one "prototypical" example per class, the model can reach surprisingly high accuracy, effectively bridging the gap between SSL and few-shot learning.

Key Results

FixMatch set new records across the board, often outperforming much more complex models.

Performance Comparison Table
  • CIFAR-10: Achieved 94.93% accuracy with only 250 labels. Even more impressively, it reached 88.61% with just 40 labels (4 per class).
  • ImageNet: On the 10% labeled split, FixMatch achieved a top-1 error rate of 28.54%, significantly better than previous SOTA methods like UDA.
  • Barely Supervised: In an experimental "1-label-per-class" setup on CIFAR-10, FixMatch reached a median accuracy of 64.28%, and up to 78% when the labeled examples were particularly representative of their classes.

Limitations and Future Work

  • Label Quality Sensitivity: In the extreme low-label regime (1 label per class), performance is highly dependent on which image is labeled. If the labeled image is an outlier, the model struggles.
  • CIFAR-100 Performance: While FixMatch is excellent, it only beat the previous SOTA (ReMixMatch) on CIFAR-100 when the authors added back a "Distribution Alignment" component. This suggests that for datasets with many classes, additional mechanisms to ensure the model doesn't ignore certain categories may still be necessary.
  • Hyperparameter Tuning: Although simpler than its predecessors, FixMatch still relies on a manually set confidence threshold ($\tau = 0.95$), which might need tuning for different domains like medical imaging or audio.

Summary updated

Abstract

Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model’s performance. This domain has seen fast progress recently, at the cost of requiring more complex methods. In this paper we propose FixMatch, an algorithm that is a significant simplification of existing SSL methods. FixMatch first generates pseudo-labels using the model’s predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 – just 4 labels per class. We carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch’s success. The code is available at https://github.com/google-research/fixmatch.
BibTeX
@inproceedings{sohn2020fixmatch,
  author = {Kihyuk Sohn and David Berthelot and Nicholas Carlini and Zizhao Zhang and Han Zhang and Colin Raffel and Ekin Dogus Cubuk and Alexey Kurakin and Chun-Liang Li},
  title = {{FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=NkfFyw3KIr},
}
Figure