Deep learning is notoriously data-hungry, yet human-annotated labels remain expensive and time-consuming to acquire. Semi-supervised learning (SSL) offers a compelling solution by leveraging a small amount of labeled data alongside a massive corpus of unlabeled data. A particularly successful technique in this domain is pseudo-labeling: training a "teacher" model on available labels to generate predictions for the unlabeled data, and subsequently training a "student" model on this combined dataset.
While pseudo-labeling achieves state-of-the-art performance in practice—such as in complex image classification tasks—the mathematical mechanisms explaining why it works for neural networks have remained largely elusive. Previous theoretical understandings were mostly confined to simple linear models. This paper, presented at ICLR 2023, bridges this gap by providing a rigorous theoretical analysis of the training dynamics of a two-layer Convolutional Neural Network (CNN), revealing exactly why SSL succeeds where standard supervised learning fails.
The Core Mechanism: Feature Learning vs. Noise Memorization
The authors demonstrate that SSL with pseudo-labelers succeeds because it forces neural networks to extract meaningful features from a sea of unlabeled data, even when the pseudo-labels are noisy.
In a standard supervised learning (SL) setup with very few labels, a CNN is easily distracted by high-variance noise. To drive the training loss to zero, the model simply "memorizes" this noise rather than learning the underlying signal. However, in the SSL setting, the true signal remains consistent across thousands of unlabeled images, while the noise varies from image to image. Even if the pseudo-labels provided by the teacher model are only slightly better than a random guess (e.g., 51% accuracy), the sheer volume of unlabeled data allows the student model to "purify" the signal and learn the correct features.
Methodology and The Data Model
To prove this mathematically, the authors employ a controlled "case study" approach, comparing SSL and SL within a specific mathematical environment:
- The Toy Dataset: They design a synthetic high-dimensional dataset ($d = 10,000$) where each image consists of a signal patch (which dictates the true class) and a noise patch (random Gaussian noise orthogonal to the signal). Crucially, the noise has a larger norm than the signal, making it a tempting distraction for the network.
- Data Scarcity: The setup mimics a "low-label, high-noise" regime, utilizing only 20 labeled samples ($n_l = 20$) compared to 20,000 unlabeled samples ($n_u = 20,000$).
- The Model: A two-layer CNN where filters are applied to the patches.
The semi-supervised pipeline analyzed consists of two distinct phases:
- Pre-training: A pseudo-labeler provides noisy labels for the large unlabeled dataset. The student CNN is trained on this data, allowing its convolutional filters to learn the core feature representations.
- Linear Probing: The student's learned convolutional filters are frozen, and a simple linear classification layer is trained exclusively on the small, high-quality labeled dataset.
By utilizing a "beyond-NTK" (Neural Tangent Kernel) perspective, the authors track how the model's features actively evolve during Gradient Descent, rather than assuming the model remains close to its initialization state.
Key Theoretical Results
The paper establishes a formal mathematical "separation" between the capabilities of SSL and SL under these conditions:
- SSL Success (Theorems 4.2 & 4.3): After pre-training on the pseudo-labeled data and fine-tuning via linear probing on just 20 true labels, the model successfully learns the underlying signal and achieves nearly zero test error.
- SL Failure (Theorem 4.4): When standard supervised learning is applied directly to those same 20 labeled examples, the model perfectly overfits the training set by memorizing noise. Consequently, it fails to generalize, resulting in a test loss of $\Theta(1)$ (performing essentially no better than random chance).
These theoretical proofs are perfectly matched by the authors' experimental simulations. In the SSL model, metrics tracking "feature learning" shoot up while "noise memorization" remains low. For the SL model, the exact opposite occurs.
Limitations and Future Directions
While this paper provides a crucial foundational understanding of pseudo-labeling, the authors note some limitations inherent to their theoretical framework. The toy data model assumes the signal and noise are perfectly orthogonal, which is a simplification of real-world image data. Additionally, the study focuses on a two-layer CNN with a fixed second layer, whereas modern architectures like ResNets are significantly deeper and train all layers simultaneously.
Future work will look to extend this analysis to "linearly non-separable" data distributions and explore more complex network architectures, such as those where the convolutional stride is smaller than the filter size.
Reference: Kou, Y., Chen, Z., Cao, Y., & Gu, Q. (2023). How Does Semi-supervised Learning with Pseudo-labelers Work? A Case Study. Published as a conference paper at ICLR 2023.
Summary updated