TL;DR
PSPU (Pseudo-Supervised PU learning) is a new framework that solves the "overfitted risk estimation" problem in Positive and Unlabeled (PU) learning. By iteratively generating high-quality pseudo-labels and training with standard semi-supervised objectives, it achieves state-of-the-art performance, particularly in challenging imbalanced scenarios like industrial anomaly detection.
What Problem Does This Paper Address?
In many real-world scenarios—like fraud detection, medical diagnosis, or industrial anomaly detection—we have plenty of data, but only a tiny fraction is labeled as "positive" (e.g., a known defect). The rest is "unlabeled," containing a mix of both positive and negative samples. This is the domain of Positive and Unlabeled (PU) learning.
Traditional PU learning methods rely on a "risk estimator" to guess which unlabeled samples are negative. However, these estimators often suffer from overfitted risk estimation. They assume that the small set of labeled positive samples perfectly represents the distribution of all positive samples (the SCAR assumption). In practice, this assumption fails; for example, a few photos of deer in one posture might not help a model recognize deer in another posture within the unlabeled pool. As shown in Figure 1, this gap leads the model to misclassify unlabeled positives as negatives.
Methods
The authors propose PSPU, a two-stage iterative framework designed to bypass the biases of traditional PU risk estimators:
- Pseudo Supervision Generation: In each epoch, a basic PU network (like nnPU) identifies the most confident positive and negative samples from the unlabeled pool. To handle potential mislabeling, they use a supervision mixture mechanism based on Mixup, interpolating these confident samples to create a "smoother" and more robust training set.
- Pseudo-supervised Training: A second network is trained using standard Semi-Supervised Learning (SSL) objectives (like FixMatch or MixMatch). Because these objectives don't rely on the biased PU risk estimator, they can better correct the model's weights. They also include a feature consistency loss ($L_{con}$) to ensure the model is robust to noise.
- Progressive Knowledge Transfer (PKT): Instead of just replacing the PU model with the new one, the framework uses a weighted average to merge the weights of the two networks. This "progressive" feedback loop prevents the model from over-correcting and helps it converge more stably.
Datasets
The researchers evaluated PSPU across several benchmarks:
- Standard Vision Datasets: MNIST, CIFAR-10, and CIFAR-100.
- Industrial Anomaly Detection: MVTecAD, which represents a highly imbalanced, real-world application where defects (positives) are rare.
- Imbalanced Settings: They specifically tested "extremely imbalanced" versions of CIFAR where the ratio of positive to negative samples is heavily skewed.
Key Insights and Ideas
- The Bias Gap: The authors empirically prove that the gap between estimated risk and "oracle" (true) risk widens as training progresses, especially when data is imbalanced. This confirms that overfitting to the small labeled set is a primary failure mode for PU learning.
- SSL as a Corrective: Standard semi-supervised learning objectives are often more powerful than PU-specific ones, provided you can generate reliable pseudo-labels. PSPU acts as a bridge, using PU learning to "bootstrap" the pseudo-labels needed for SSL.
- Mixup for Denoising: While Mixup is usually for data augmentation, here it serves a critical role in mitigating the impact of noisy pseudo-labels by creating a continuous supervision signal rather than hard (and potentially wrong) labels.
- Stability through PKT: Direct weight transfer can lead to erratic training. The Progressive Knowledge Transfer ensures that the "knowledge" gained from the pseudo-supervised stage is integrated smoothly back into the main model.
Key Results
PSPU consistently outperformed existing state-of-the-art (SOTA) methods:
- CIFAR-10 (Imbalanced): Achieved an 80.16% F1 score, a massive ~17% improvement over the previous best method (HolisticPU at 67.20%).
- MVTecAD: Improved the F1 score by 11% compared to HolisticPU, demonstrating its effectiveness in real-world industrial settings.
- Robustness: In "extremely imbalanced" cases where traditional methods like nnPU saw their F1 scores drop to zero (meaning they classified everything as negative), PSPU remained stable and effective.
Limitations and Future Work
While the paper demonstrates strong results, a few areas remain for exploration:
- Prior Knowledge: Like most PU methods, PSPU assumes the prior probability of positive samples ($\pi_p$) is known. In some wild scenarios, this might be hard to estimate accurately.
- Computational Overhead: The two-stage iterative process and weight merging add complexity compared to a single-stage model.
- Future Directions: The authors suggest that the framework is versatile and could be integrated with even more advanced SSL objectives or different backbone architectures in the future.
Summary updated