Robust Deep Semi-Supervised Learning: A Brief Introduction

Lan-Zhe Guo, Zhi Zhou, Yu-Feng LiarXiv 2022 · OpenReview

TL;DR

This paper surveys the critical but often overlooked field of robust semi-supervised learning (SSL). It introduces a formal definition of robust SSL and categorizes common threats to unlabeled data (distribution, feature, and label corruption), providing a unified framework for understanding and addressing these vulnerabilities.

What Problem Does This Paper Address?

Traditional semi-supervised learning (SSL) excels at improving model performance by leveraging abundant unlabeled data when labeled data is scarce. However, this success hinges on the assumption that unlabeled data is "perfect" – meaning it comes from the same distribution as labeled data and is free from corruption. In real-world applications, this assumption often breaks down. Unlabeled data can be corrupted in various ways: its distribution might differ from labeled data, its features might be maliciously altered (adversarial attacks), or its class labels might be imbalanced. When SSL algorithms encounter such corrupted data, their performance can degrade significantly, undermining their practical utility. This paper addresses the urgent need for SSL methods that are robust to these real-world data imperfections.

Methods

This paper is primarily a survey and taxonomy of existing work on robust SSL, rather than introducing new algorithms. The authors' main methodological contribution is the formalization and classification of robustness threats. They define robust SSL as an SSL algorithm whose performance does not degrade severely when unlabeled data is corrupted. They then categorize these corruption types into three main groups:

  • Distribution Corruption: The distribution of unlabeled data ($p_u(x, y)$) differs from the distribution of labeled data ($p_l(x, y)$). This is further broken down into:
  • Feature Distribution Mismatch: The feature distributions ($p_l(x)$ vs. $p_u(x)$) differ.
  • Label Distribution Mismatch: The label distributions ($p_l(y)$ vs. $p_u(y)$) differ, often manifesting as class imbalance or open-set recognition problems.
  • Feature Corruption: The features of unlabeled examples are perturbed, most notably through adversarial attacks.
  • Label Corruption: The distribution of labels in the unlabeled data is imbalanced (e.g., a long-tailed distribution).

The paper then reviews existing SSL techniques that aim to address these specific corruption types, categorizing them by their approach (e.g., feature adaptation, example re-weighting, adversarial defense, pseudo-label alignment).

Datasets

The paper discusses the use of standard benchmark datasets like CIFAR-10 and Tiny ImageNet for simulating corruption scenarios (e.g., creating out-of-distribution data). It also mentions real-world datasets like ImageNet, WebVision, and medical imaging datasets as examples where these corruption issues naturally arise. The authors highlight the need for new benchmark datasets that better reflect realistic unlabeled data corruption.

Key Insights and Ideas

  • The "Perfect Unlabeled Data" Illusion: Real-world unlabeled data is rarely perfect. This fundamental gap between benchmark assumptions and practical reality is a major hurdle for deploying SSL.
  • A Unified Taxonomy is Crucial: By categorizing corruption into distribution, feature, and label issues, the paper provides a clear framework for understanding the diverse challenges in robust SSL and for organizing research efforts.
  • Adversarial Robustness is More Critical for SSL: While adversarial robustness is important for supervised learning, it's even more so for SSL because adversaries can more easily "poison" the large pool of unlabeled data.
  • SSL's Vulnerability to Imbalance: Standard SSL methods can exacerbate class imbalance issues, performing worse than supervised methods on minority classes by generating biased pseudo-labels.
  • The Need for Realistic Benchmarks: Current simulated corruption might not fully capture the complexity of real-world data issues, leading to models that overfit to specific benchmark artifacts rather than truly robust solutions.

Key Results

As this is a survey paper, it doesn't present new quantitative results from experiments. Instead, it synthesizes the findings of numerous prior works. The key "results" are the insights derived from this synthesis:

  • SSL algorithms are indeed vulnerable to various forms of unlabeled data corruption, leading to significant performance degradation.
  • Existing research has developed various strategies to mitigate these issues, broadly falling into categories like feature adaptation, example re-weighting, adversarial defense, and pseudo-label alignment.
  • Despite progress, robust SSL is still in its infancy, with many open challenges and promising future research directions.

Limitations and Future Work

The authors acknowledge that robust SSL is a complex and evolving field. They identify several promising future research directions:

  • Robustness to General Data Types: Developing SSL methods that work well on heterogeneous tabular data, which lacks the inherent structure of images or text.
  • Robustness to Compound Weakly Supervised Data: Extending robust SSL to scenarios where unlabeled data is not only corrupted but also comes with other forms of weak supervision (e.g., inexact or inaccurate labels).
  • Robust SSL with Domain Knowledge: Integrating symbolic domain knowledge (e.g., rules) with unlabeled data to improve robustness.
  • Learning in Dynamic Environments: Creating SSL algorithms that can adapt to changing data distributions over time, a common challenge in real-world deployments.
  • Building Realistic Data Sets: The critical need for new, realistic benchmark datasets that accurately reflect unlabeled data corruption to drive progress in the field.

Summary updated

Abstract

Semi-supervised learning (SSL) is the branch of machine learning that aims to improve learning performance by leveraging unlabeled data when labels are insufficient. Recently, SSL with deep models has proven to be successful on standard benchmark tasks. However, they are still vulnerable to various robustness threats in real-world applications as these benchmarks provide perfect unlabeled data, while in realistic scenarios, unlabeled data could be corrupted. Many researchers have pointed out that after exploiting corrupted unlabeled data, SSL suffers severe performance degradation problems. Thus, there is an urgent need to develop SSL algorithms that could work robustly with corrupted unlabeled data. To fully understand robust SSL, we conduct a survey study. We first clarify a formal definition of robust SSL from the perspective of machine learning. Then, we classify the robustness threats into three categories: i) distribution corruption, i.e., unlabeled data distribution is mismatched with labeled data; ii) feature corruption, i.e., the features of unlabeled examples are adversarially attacked; and iii) label corruption, i.e., the label distribution of unlabeled data is imbalanced. Under this unified taxonomy, we provide a thorough review and discussion of recent works that focus on these issues. Finally, we propose possible promising directions within robust SSL to provide insights for future research.
BibTeX
@misc{guo2022robust,
  author = {Lan-Zhe Guo and Zhi Zhou and Yu-Feng Li},
  title = {{Robust Deep Semi-Supervised Learning: A Brief Introduction}},
  year = {2022},
  eprint = {2202.05975},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.05975},
}
Figure