TL;DR
This paper introduces FUSSL, a novel hierarchical training protocol for self-supervised learning (SSL) that combines sample-level and cluster-level supervision. By leveraging fuzzy pseudo-labeling, FUSSL aims to improve the robustness and performance of existing SSL methods, demonstrating consistent gains across various baselines.
What Problem Does This Paper Address?
Self-supervised learning (SSL) has become a powerful paradigm for learning representations from unlabeled data, aiming to match or exceed supervised learning performance. However, a key challenge for SSL methods is their robustness across different settings and downstream tasks. The authors identify that current SSL approaches often rely on a single type of supervisory signal (either sample-level invariance to augmentations or cluster-level assignments), which can lead to limitations. Sample-level supervision can sometimes degrade performance on downstream tasks, while cluster-level supervision can be computationally expensive and less generalizable. FUSSL seeks to address this by creating a more robust and generalizable SSL framework through a dual-supervision approach.
Methods
FUSSL proposes a two-phase hierarchical training protocol that can be applied to any existing SSL baseline.
- Phase 1: Sample-Level Invariance Enforcement: This phase leverages the standard SSL approach of enforcing invariance to data augmentations. An ensemble of models is trained, where each model observes its own set of augmented views of a sample. This ensemble approach, combined with the inherent uncertainty from multiple augmented views and model variations, is designed to generate more robust representations.
- Phase 2: Fuzzy Pseudo-Labeling and Cluster-Level Learning: After Phase 1, the learned representations are used to generate "fuzzy" pseudo-labels for the original samples. This involves a combination of hard and soft labels, where soft labels allow for uncertainty and cross-class feature learning. A single model from the ensemble is then trained using these pseudo-labels, focusing on learning medium to high-granularity features associated with class information. This phase aims to learn cross-class features, enhancing generalization and robustness. The "fuzzy" aspect comes from the probabilistic nature of the pseudo-labels, allowing for a more nuanced representation than strict hard labels.
The core idea is to decompose the Information Bottleneck principle into two parts, each guided by a distinct supervisory signal, and to minimize the mutual information between these two signals to encourage diverse feature learning.
Datasets
The paper uses the following datasets:
- ImageNet: Used for pre-training the SSL baselines.
- CIFAR10 and CIFAR100: Used for transfer learning evaluation.
- Tiny ImageNet: Used for ablation studies.
Key Insights and Ideas
- Dual Supervision is Key: The paper argues that relying on a single supervisory signal in SSL is a fundamental limitation. Combining sample-level and cluster-level supervision, as FUSSL does, offers a more comprehensive approach.
- Fuzzy Pseudo-Labeling for Robustness: The introduction of "fuzzy" pseudo-labeling (using both hard and soft labels) in the second phase is crucial. This allows the model to learn more nuanced, cross-class features, leading to better generalization and robustness compared to using only hard pseudo-labels.
- Ensemble for Uncertainty: Using an ensemble of models in the first phase injects uncertainty into the learning process, which is then leveraged for fuzzy pseudo-labeling. This ensemble approach helps explore the representation space more effectively.
- Generalizability: FUSSL is designed to be a general protocol applicable to various SSL baselines, not tied to specific architectural designs or pretext tasks.
Key Results
- Consistent Improvement: FUSSL consistently improves the top-1 linear classification accuracy of various SSL baselines on ImageNet, with improvements ranging from 0.8% to 1.4%.
- Transfer Learning Gains: The protocol also shows benefits in transfer learning scenarios on CIFAR10 and CIFAR100, improving performance across baselines.
- Ablation Study Findings:
- Soft pseudo-labeling generally yields better results than hard pseudo-labeling.
- An ensemble size of 3 (m=3) provided the highest average improvement.
- Progressive pseudo-labeling (iterative phase 1/phase 2 training) showed a drop in improvement, suggesting the effectiveness of the proposed sequential strategy.
- Fixing early layers of the backbone in Phase 2 (conditional learning) significantly improves performance compared to retraining all layers.
Limitations and Future Work
- Lower Gains for Clustering Methods: The paper notes that FUSSL shows lower improvements for clustering-based approaches like SwAV. This is attributed to SwAV already learning medium to high-granularity features, making the additional supervision less impactful.
- Backdoor Attack Robustness: While not extensively detailed, the authors mention that FUSSL with m > 1 can improve robustness against backdoor attacks on SSL under careful settings. This suggests a promising avenue for future research in adversarial robustness.
- Future Work: The authors suggest exploring further scenarios for fuzzy pseudo-labeling and investigating the impact of different ensemble sizes and training schedules. The potential for FUSSL to enhance robustness against various forms of data corruption and adversarial attacks is also highlighted.
Summary updated