Towards Discovering the Effectiveness of Moderately Confident Samples for Semi-Supervised Learning

Hui Tang, Kui Jia

Introduction

Semi-supervised learning (SSL) has emerged as a critical paradigm for training deep neural networks when labeled data is scarce. By leveraging a small set of labeled examples alongside a vast pool of unlabeled data, SSL aims to achieve performance comparable to fully supervised methods. A cornerstone of modern SSL is "self-training," where a model generates "pseudo-labels" for unlabeled data to guide its own learning process.

The Challenge of Confidence Thresholds

To maintain the quality of pseudo-labels, state-of-the-art frameworks like FixMatch employ a high confidence threshold (typically $\tau = 0.95$). Only samples where the model's prediction exceeds this threshold are used for training. While this "cleanliness-first" approach prevents the model from learning from incorrect labels, it leads to significant data waste. Specifically, "moderately confident" samples—those with confidence scores between 0.75 and 0.95—are discarded, despite potentially containing valuable information for refining decision boundaries.

The authors of this work propose the Taylor Expansion Inspired Filtration (TEIF) framework to identify and utilize these informative "diamonds in the rough" without compromising training stability.

The TEIF Framework

The TEIF framework is grounded in the mathematical principle of Taylor expansion. By analyzing the loss function's expansion, the authors derive conditions under which adding a new sample to the training set ensures a stable and informative update to the network's weights. From this theoretical foundation, two practical filtration mechanisms are developed:

Gradient Synchronization Filter (GSF)

The GSF operates on the principle of directional alignment in the optimization landscape. It identifies a "majority gradient" direction based on the most reliable data (labeled and high-confidence unlabeled samples). A moderately confident sample is selected only if its gradient aligns—or "synchronizes"—with this majority direction.

Conceptual illustration of the Gradient Synchronization Filter (GSF), showing how samples are selected based on gradient alignment with the decision boundary.

Prototype Proximity Filter (PPF)

Complementing the gradient-based approach, the PPF focuses on the feature representation space. It maintains "prototypes" for each class, which are the average feature vectors of high-confidence samples. A moderately confident sample is deemed reliable if it resides within a specific proximity region of a class prototype, suggesting it is a representative example of that category.

Conceptual illustration of the Prototype Proximity Filter (PPF), highlighting the selection of samples within the proximity region of class prototypes.

Experimental Results

The TEIF framework was integrated into the FixMatch baseline and evaluated on standard benchmarks including CIFAR-10, CIFAR-100, and SVHN. The results demonstrate substantial improvements, particularly in extremely low-label regimes:

  • CIFAR-10 (40 labels): The error rate dropped from 13.81% (FixMatch) to 7.71%.
  • CIFAR-100 (400 labels): Error rates were reduced from 48.85% to 41.96%.
  • SVHN (40 labels): Performance improved from a 3.96% error rate to 3.10%.

Beyond final accuracy, the filters also enhanced training efficiency, allowing models to reach high performance levels earlier in the training cycle by utilizing a larger portion of the unlabeled pool from the outset.

Discussion and Limitations

While TEIF significantly pushes the boundaries of SSL, it is not without limitations. The current implementation relies on a first-order Taylor approximation to maintain computational efficiency. While effective, higher-order terms (such as the Hessian) might offer even more precise filtration at a higher computational cost. Additionally, the framework introduces new hyperparameters, such as the synchronization threshold $\tau_s$, which require careful tuning in scenarios with extremely limited labeled data.

In conclusion, this work demonstrates that moderately confident samples are a valuable resource in semi-supervised learning. By applying theoretically grounded filters like GSF and PPF, researchers can unlock the potential of discarded data to build more robust and data-efficient models.

Summary updated

Abstract

Semi-supervised learning (SSL) has been studied for a long time to solve vision tasks in data-efficient application scenarios. SSL aims to learn a good classification model using a few labeled data together with large-scale unlabeled data. Recent advances achieve the goal by combining multiple SSL techniques, e.g., self-training and consistency regularization. From unlabeled samples, they usually adopt a confidence filter (CF) to select reliable ones with high prediction confidence. In this work, we study whether the moderately confident samples are useless and how to select the useful ones to improve model optimization. To answer these problems, we propose a novel Taylor expansion inspired filtration (TEIF) framework, which admits the samples of moderate confidence with similar feature or gradient to the respective one averaged over the labeled and highly confident unlabeled data. It can produce a stable and new information induced network update, leading to better generalization. Two novel filters are derived from this framework and can be naturally explained in two perspectives. One is gradient synchronization filter (GSF), which strengthens the optimization dynamic of fully-supervised learning; it selects the samples whose gradients are similar to classwise majority gradients. The other is prototype proximity filter (PPF), which involves more prototypical samples in training to learn better semantic representations; it selects the samples near classwise prototypes. They can be integrated into SSL methods with CF. We use the state-of-the-art Fix-Match as the baseline. Experiments on popular SSL benchmarks show that we achieve the new state of the art.
BibTeX
@inproceedings{tang2022towards,
  author = {Hui Tang and Kui Jia},
  title = {{Towards Discovering the Effectiveness of Moderately Confident Samples for Semi-Supervised Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01425},
}
Figure