TL;DR
This paper surveys the critical but often overlooked field of robust semi-supervised learning (SSL). It introduces a formal definition of robust SSL and categorizes common threats to unlabeled data (distribution, feature, and label corruption), providing a unified framework for understanding and addressing these vulnerabilities.
What Problem Does This Paper Address?
Traditional semi-supervised learning (SSL) excels at improving model performance by leveraging abundant unlabeled data when labeled data is scarce. However, this success hinges on the assumption that unlabeled data is "perfect" – meaning it comes from the same distribution as labeled data and is free from corruption. In real-world applications, this assumption often breaks down. Unlabeled data can be corrupted in various ways: its distribution might differ from labeled data, its features might be maliciously altered (adversarial attacks), or its class labels might be imbalanced. When SSL algorithms encounter such corrupted data, their performance can degrade significantly, undermining their practical utility. This paper addresses the urgent need for SSL methods that are robust to these real-world data imperfections.
Methods
This paper is primarily a survey and taxonomy of existing work on robust SSL, rather than introducing new algorithms. The authors' main methodological contribution is the formalization and classification of robustness threats. They define robust SSL as an SSL algorithm whose performance does not degrade severely when unlabeled data is corrupted. They then categorize these corruption types into three main groups:
- Distribution Corruption: The distribution of unlabeled data ($p_u(x, y)$) differs from the distribution of labeled data ($p_l(x, y)$). This is further broken down into:
- Feature Distribution Mismatch: The feature distributions ($p_l(x)$ vs. $p_u(x)$) differ.
- Label Distribution Mismatch: The label distributions ($p_l(y)$ vs. $p_u(y)$) differ, often manifesting as class imbalance or open-set recognition problems.
- Feature Corruption: The features of unlabeled examples are perturbed, most notably through adversarial attacks.
- Label Corruption: The distribution of labels in the unlabeled data is imbalanced (e.g., a long-tailed distribution).
The paper then reviews existing SSL techniques that aim to address these specific corruption types, categorizing them by their approach (e.g., feature adaptation, example re-weighting, adversarial defense, pseudo-label alignment).
Datasets
The paper discusses the use of standard benchmark datasets like CIFAR-10 and Tiny ImageNet for simulating corruption scenarios (e.g., creating out-of-distribution data). It also mentions real-world datasets like ImageNet, WebVision, and medical imaging datasets as examples where these corruption issues naturally arise. The authors highlight the need for new benchmark datasets that better reflect realistic unlabeled data corruption.
Key Insights and Ideas
- The "Perfect Unlabeled Data" Illusion: Real-world unlabeled data is rarely perfect. This fundamental gap between benchmark assumptions and practical reality is a major hurdle for deploying SSL.
- A Unified Taxonomy is Crucial: By categorizing corruption into distribution, feature, and label issues, the paper provides a clear framework for understanding the diverse challenges in robust SSL and for organizing research efforts.
- Adversarial Robustness is More Critical for SSL: While adversarial robustness is important for supervised learning, it's even more so for SSL because adversaries can more easily "poison" the large pool of unlabeled data.
- SSL's Vulnerability to Imbalance: Standard SSL methods can exacerbate class imbalance issues, performing worse than supervised methods on minority classes by generating biased pseudo-labels.
- The Need for Realistic Benchmarks: Current simulated corruption might not fully capture the complexity of real-world data issues, leading to models that overfit to specific benchmark artifacts rather than truly robust solutions.
Key Results
As this is a survey paper, it doesn't present new quantitative results from experiments. Instead, it synthesizes the findings of numerous prior works. The key "results" are the insights derived from this synthesis:
- SSL algorithms are indeed vulnerable to various forms of unlabeled data corruption, leading to significant performance degradation.
- Existing research has developed various strategies to mitigate these issues, broadly falling into categories like feature adaptation, example re-weighting, adversarial defense, and pseudo-label alignment.
- Despite progress, robust SSL is still in its infancy, with many open challenges and promising future research directions.
Limitations and Future Work
The authors acknowledge that robust SSL is a complex and evolving field. They identify several promising future research directions:
- Robustness to General Data Types: Developing SSL methods that work well on heterogeneous tabular data, which lacks the inherent structure of images or text.
- Robustness to Compound Weakly Supervised Data: Extending robust SSL to scenarios where unlabeled data is not only corrupted but also comes with other forms of weak supervision (e.g., inexact or inaccurate labels).
- Robust SSL with Domain Knowledge: Integrating symbolic domain knowledge (e.g., rules) with unlabeled data to improve robustness.
- Learning in Dynamic Environments: Creating SSL algorithms that can adapt to changing data distributions over time, a common challenge in real-world deployments.
- Building Realistic Data Sets: The critical need for new, realistic benchmark datasets that accurately reflect unlabeled data corruption to drive progress in the field.
Summary updated