RegMixMatch: Optimizing Mixup Utilization in Semi-Supervised Learning

Haorong Han, Jidong Yuan, Chixuan Wei, Zhongyang YuAAAI 2025 · DOI

TL;DR

This paper introduces RegMixMatch, a novel semi-supervised learning (SSL) framework that optimizes the use of Mixup, a data augmentation technique, by addressing its tendency to reduce label purity. RegMixMatch achieves state-of-the-art performance by intelligently incorporating both high- and low-confidence unlabeled data, leading to more robust and accurate models.

What Problem Does This Paper Address?

Semi-supervised learning (SSL) aims to train models effectively using a small amount of labeled data and a large amount of unlabeled data. A common strategy in SSL is to use pseudo-labels generated from model predictions on unlabeled data. Data augmentation techniques, particularly Mixup, have been widely adopted to improve model generalization and consistency regularization. However, this paper identifies a critical issue: Mixup, while beneficial in supervised settings, can actually degrade SSL performance by reducing the purity of pseudo-labels. This happens because Mixup interpolates between samples and their labels, leading to less confident and potentially noisy artificial labels. Existing methods that filter out low-confidence pseudo-labels, while mitigating confirmation bias, also discard potentially valuable information. The core problem is how to effectively utilize both high- and low-confidence unlabeled data in SSL, especially when using Mixup, without sacrificing label purity.

Methods

RegMixMatch tackles this problem with two main innovations:

  • Semi-supervised RegMixup (SRM): This component adapts the RegMixup technique for SSL. It leverages pseudo-labeling to retain high-confidence predictions for unlabeled data. Crucially, it uses both clean (high-confidence pseudo-labeled) and mixed samples (generated via Mixup from high-confidence samples) for training. This approach aims to mitigate the high-entropy behavior of Mixup that reduces label purity by carefully balancing the use of clean and mixed data.
  • Class-Aware Mixup (CAM): This technique is designed to effectively utilize low-confidence unlabeled samples, which are typically discarded. Instead of randomly mixing these samples, CAM mixes them with high-confidence samples that share the same predicted class. This class-aware mixing helps to reduce noise in the artificial labels of low-confidence samples by incorporating relevant features from the predicted class. Furthermore, CAM leverages the observation that the top-2 predicted classes often contain the true label, making the inclusion of this information beneficial.

RegMixMatch combines these two components, along with a standard supervised loss on labeled data, to form its overall objective function.

Datasets

The paper evaluates RegMixMatch on several standard SSL benchmarks:

  • CIFAR-10
  • CIFAR-100
  • SVHN
  • STL-10
  • ImageNet

The experiments are conducted under various labeled data conditions, including settings with as few as 10 labels for CIFAR-10 and 40 labels for STL-10.

Key Insights and Ideas

  • Mixup's Double-Edged Sword in SSL: While Mixup is a powerful tool for supervised learning, its application in SSL can be detrimental due to its tendency to reduce the purity of pseudo-labels, leading to worse performance than simpler methods.
  • The Value of Low-Confidence Samples: Contrary to common practice, low-confidence samples are not entirely useless. By intelligently incorporating them (as done by CAM), their potential can be unlocked, improving overall performance.
  • Class-Awareness is Key for Low-Confidence Data: Randomly mixing low-confidence samples can introduce more noise. Mixing them with high-confidence samples of the same predicted class is a more effective strategy to improve label quality.
  • Thresholding for Mixup Matters: Using different thresholds for retaining pseudo-labels for consistency loss ($T_c$) versus for Mixup ($T_m$) is crucial. A higher threshold for Mixup ($T_m$) helps ensure that only truly high-confidence samples are used for mixing, preserving label purity.

Key Results

RegMixMatch achieves state-of-the-art (SOTA) performance in 9 out of 12 commonly used SSL scenarios.

  • On CIFAR-10 with 10 labels, RegMixMatch achieved an error rate of 4.35%, surpassing the second-best by 3.72%.
  • On STL-10 with 40 labels, it achieved an error rate of 11.74%, outperforming the second-best by 3.71%.
  • On ImageNet with 10,000 labels, RegMixMatch showed a 3.66% improvement over other methods.
  • An efficiency study shows RegMixMatch achieves substantial performance gains with only a minimal increase in runtime compared to other advanced SSL methods.

Limitations and Future Work

  • Hyperparameter Sensitivity: The paper acknowledges that RegMixMatch's performance is sensitive to hyperparameters like the confidence threshold for Mixup ($T_m$) and mixing intensities ($a_h, a_l$). While optimal values are found, further research into more adaptive or automated hyperparameter tuning could be beneficial.
  • Future Research Direction: The authors suggest that RegMixMatch represents an "initial exploration into the potential of low-confidence samples," implying that further research into novel ways to leverage these challenging samples is warranted. They also recommend further studies on the optimal mixing strategies and their impact on model robustness.

Summary updated

BibTeX
@inproceedings{han2025regmixmatch,
  author = {Haorong Han and Jidong Yuan and Chixuan Wei and Zhongyang Yu},
  title = {{RegMixMatch: Optimizing Mixup Utilization in Semi-Supervised Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i16.33872},
}
Figure