Introduction
Semi-supervised learning (SSL) has emerged as a critical paradigm for training deep neural networks when labeled data is scarce. By leveraging a small set of labeled examples alongside a vast pool of unlabeled data, SSL aims to achieve performance comparable to fully supervised methods. A cornerstone of modern SSL is "self-training," where a model generates "pseudo-labels" for unlabeled data to guide its own learning process.
The Challenge of Confidence Thresholds
To maintain the quality of pseudo-labels, state-of-the-art frameworks like FixMatch employ a high confidence threshold (typically $\tau = 0.95$). Only samples where the model's prediction exceeds this threshold are used for training. While this "cleanliness-first" approach prevents the model from learning from incorrect labels, it leads to significant data waste. Specifically, "moderately confident" samples—those with confidence scores between 0.75 and 0.95—are discarded, despite potentially containing valuable information for refining decision boundaries.
The authors of this work propose the Taylor Expansion Inspired Filtration (TEIF) framework to identify and utilize these informative "diamonds in the rough" without compromising training stability.
The TEIF Framework
The TEIF framework is grounded in the mathematical principle of Taylor expansion. By analyzing the loss function's expansion, the authors derive conditions under which adding a new sample to the training set ensures a stable and informative update to the network's weights. From this theoretical foundation, two practical filtration mechanisms are developed:
Gradient Synchronization Filter (GSF)
The GSF operates on the principle of directional alignment in the optimization landscape. It identifies a "majority gradient" direction based on the most reliable data (labeled and high-confidence unlabeled samples). A moderately confident sample is selected only if its gradient aligns—or "synchronizes"—with this majority direction.
Prototype Proximity Filter (PPF)
Complementing the gradient-based approach, the PPF focuses on the feature representation space. It maintains "prototypes" for each class, which are the average feature vectors of high-confidence samples. A moderately confident sample is deemed reliable if it resides within a specific proximity region of a class prototype, suggesting it is a representative example of that category.
Experimental Results
The TEIF framework was integrated into the FixMatch baseline and evaluated on standard benchmarks including CIFAR-10, CIFAR-100, and SVHN. The results demonstrate substantial improvements, particularly in extremely low-label regimes:
- CIFAR-10 (40 labels): The error rate dropped from 13.81% (FixMatch) to 7.71%.
- CIFAR-100 (400 labels): Error rates were reduced from 48.85% to 41.96%.
- SVHN (40 labels): Performance improved from a 3.96% error rate to 3.10%.
Beyond final accuracy, the filters also enhanced training efficiency, allowing models to reach high performance levels earlier in the training cycle by utilizing a larger portion of the unlabeled pool from the outset.
Discussion and Limitations
While TEIF significantly pushes the boundaries of SSL, it is not without limitations. The current implementation relies on a first-order Taylor approximation to maintain computational efficiency. While effective, higher-order terms (such as the Hessian) might offer even more precise filtration at a higher computational cost. Additionally, the framework introduces new hyperparameters, such as the synchronization threshold $\tau_s$, which require careful tuning in scenarios with extremely limited labeled data.
In conclusion, this work demonstrates that moderately confident samples are a valuable resource in semi-supervised learning. By applying theoretically grounded filters like GSF and PPF, researchers can unlock the potential of discarded data to build more robust and data-efficient models.
Summary updated