TL;DR
This paper challenges the common practice of matching unlabelled data semantically to labelled data in semi-supervised learning (SSDL), showing it can actually hurt performance. Instead, they propose using feature-space dissimilarity measures from a generic classifier as a more reliable criterion for selecting unlabelled data, especially under non-IID conditions.
What Problem Does This Paper Address?
Semi-supervised deep learning (SSDL) is a powerful technique for training models when labeled data is scarce. A common heuristic is to select unlabelled data that is "semantically similar" to the labeled data. For instance, if you have labeled images of digits, you'd ideally pair them with unlabelled images of digits, not cars. This is known as semantic data set matching. However, the authors question the effectiveness of this heuristic, particularly when the distributions of labeled and unlabelled data don't match (non-IID data), which is a common scenario in real-world applications. They hypothesize that this semantic matching might not always lead to performance gains and could even degrade it.
Methods
The core of this work involves a comprehensive simulation sandbox called non-IID-SSDL. This sandbox allows researchers to systematically test SSDL algorithms under various degrees of distribution mismatch between labeled and unlabelled data. The sandbox offers flexibility in choosing the base task (labeled data), the type of out-of-distribution (OOD) data, the source of OOD data, the proportion of OOD data, and the number of labeled samples.
The authors specifically stress-test the state-of-the-art SSDL algorithm MixMatch within this sandbox. To evaluate the effectiveness of data selection, they compare semantic matching with a novel approach: using dissimilarity measures in the feature space of a generic classifier. They employ two types of dissimilarity measures: Minkowski-based distances (Euclidean and Manhattan) and non-parametric density-based measures (Jensen-Shannon divergence and cosine distance). These measures are computed using a pre-trained Wide-ResNet on ImageNet, making the approach independent of the specific SSDL model being trained.
Datasets
The experiments utilize several standard datasets:
- MNIST: A dataset of handwritten digits.
- SVHN (Street View House Numbers): Images of house numbers.
- Tiny ImageNet: A smaller version of the ImageNet dataset.
- CIFAR-10: A dataset of 10 common object classes.
- FashionMNIST: A dataset of clothing items.
- Fashion Product: A dataset of fashion products.
- Gaussian Noise and Salt and Pepper Noise: These are synthetic datasets used to simulate noisy OOD data.
The non-IID-SSDL sandbox is a newly introduced benchmark that incorporates these datasets and various configurations for mixing them to simulate non-IID scenarios.
Key Insights and Ideas
- Semantic matching is not always optimal: The paper's central finding is that simply matching unlabelled data semantically to labelled data can sometimes degrade SSDL performance, contrary to common intuition.
- Feature-space dissimilarity is a better proxy: Dissimilarity measures computed in the feature space of a generic classifier provide a more reliable and reproducible criterion for selecting unlabelled data than semantic similarity.
- The "other half" heuristic can fail: Even when unlabelled data comes from the same source but represents different classes (e.g., half of MNIST digits vs. the other half), it doesn't guarantee better performance than using a semantically different dataset.
- Density-based measures show promise: The density-based dissimilarity measures (Jensen-Shannon and cosine distance) appear to be particularly effective proxies for predicting SSDL performance gains under non-IID conditions.
- Early selection is key: These dissimilarity measures can be computed before SSDL training, allowing for informed selection of unlabelled data to improve overall results.
Key Results
- In many cases, using either in-distribution unlabelled data or a 50-50 mix of in-distribution and OOD data outperformed the fully supervised baseline, showing the general benefit of SSDL.
- However, performance often degraded when including OOD data, especially with noisy datasets like SAPN and GN.
- Crucially, the paper demonstrates scenarios where a "different" (Dif) OOD dataset yielded better results than a "similar" (Sim) OOD dataset, directly contradicting the semantic matching heuristic. For instance, using Tiny ImageNet as OOD data for CIFAR-10 sometimes outperformed using the other half of CIFAR-10.
- The dissimilarity measures, particularly the cosine distance ($d_c$), showed a strong correlation with SSDL accuracy. The top two $d_c$ selections correctly identified the best non-IID configurations in seven out of nine cases.
- Qualitative analysis of feature distributions (Figure 1b and Table 8) supported these findings, showing that feature-space similarities/dissimilarities aligned with observed performance.
Limitations and Future Work
- Subjectivity in "Similar" and "Different": The authors acknowledge that defining what constitutes a "similar" or "different" dataset can be subjective, which is a drawback of the semantic matching heuristic itself.
- Computational Cost of Dissimilarity: While the proposed dissimilarity measures are designed to be quick, computing them on large datasets can still be computationally intensive.
- Future Work: The authors plan to further investigate the relationship between feature similarity and SSDL performance. They also see potential applications for these dissimilarity measures in data augmentation and auditing machine learning algorithms. Exploring connections to OOD detection, concept drift, and class distribution mismatch are also suggested avenues for future research.
Summary updated