Self-Supervised Learning with Ensemble Representations

Kyoungmin Han, Minsik LeeEngineering Applications of AI 2024

TL;DR

This paper introduces EnSiam, a novel self-supervised learning method that improves upon SimSiam by using ensemble representations to create more stable pseudo-labels. This leads to more robust training and state-of-the-art performance in representation learning tasks, even with challenging augmentations.

What Problem Does This Paper Address?

Self-supervised learning (SSL) has made significant strides in learning valuable data representations without human labels, approaching supervised learning performance. Contrastive learning, a popular SSL paradigm, aims to pull similar samples together and push dissimilar ones apart in an embedding space. SimSiam is a notable contrastive method known for its simplicity and effectiveness, but it suffers from sensitivity to training configurations like hyperparameters and augmentation settings, which can lead to unstable training. This paper aims to address this sensitivity issue by developing a more robust SSL method.

Methods

The core idea of EnSiam is inspired by ensemble techniques in knowledge distillation. The authors observe a similarity between contrastive SSL and the teacher-student framework in knowledge distillation, where branches without gradient flow act as "teachers" providing pseudo-labels. EnSiam enhances SimSiam by generating multiple augmented views of an input sample and then using the ensemble of these representations as a stable pseudo-label. This ensemble is formed by averaging the projected features from these augmented views.

Specifically, while SimSiam uses a single augmented view as a target for its predictor head (with a stop-gradient operation), EnSiam computes an average representation across multiple augmented views (denoted as $\hat{f}$) and uses this average as the target for all predictor heads. This ensemble approach is hypothesized to reduce variance in the pseudo-labels, leading to more stable training. The paper also provides a theoretical analysis of the gradient variance, showing that EnSiam indeed has smaller gradient variance compared to SimSiam.

Datasets

The paper evaluates EnSiam on several standard datasets for representation learning:

  • Small Datasets: CIFAR-10, CIFAR-100, STL-10, and Tiny-ImageNet.
  • Large-Scale Dataset: ImageNet-1k.
  • Transfer Learning Benchmark: Pascal VOC for object detection.

No new datasets were introduced.

Key Insights and Ideas

  • Ensemble Representations for Stability: The key insight is that averaging representations from multiple augmented views of the same image creates a more stable and reliable "pseudo-label" for the learning process, mitigating the sensitivity issues seen in methods like SimSiam.
  • Variance Reduction as the Mechanism: The paper provides a theoretical analysis suggesting that EnSiam's stability stems from reducing the variance of gradients during training, which is a common strategy for improving optimization stability.
  • Robustness to Augmentations: EnSiam demonstrates significantly less performance degradation under stronger augmentation settings compared to SimSiam variants, making it more practical for real-world scenarios where precise augmentation tuning is difficult.
  • Simplicity and Extensibility: The proposed method is a straightforward modification of SimSiam, making it easy to implement and potentially applicable to other SSL frameworks.

Key Results

  • State-of-the-Art Performance: EnSiam achieves the highest performance in linear evaluation across all tested datasets (CIFAR-10, CIFAR-100, STL-10, Tiny-ImageNet, and ImageNet).
  • Improved Transfer Learning: EnSiam shows the highest performance in transfer learning for object detection on Pascal VOC.
  • Robustness to Batch Size: EnSiam maintains stable performance across a wider range of batch sizes compared to SimSiam 4-aug, highlighting its robustness in resource-constrained environments.
  • Outperforms SimSiam Variants: EnSiam consistently outperforms SimSiam and SimSiam K-aug (with K=4 augmentations) across various experiments, including different augmentation strengths and numbers of augmentations. For instance, on Tiny-ImageNet, EnSiam (4-aug) achieves 52.43% accuracy, outperforming SimSiam 4-aug (48.95%) and SimSiam (optimized) (48.95%).

Limitations and Future Work

  • Computational Cost: Increasing the number of augmentations (K) for the ensemble can lead to higher computational and memory burdens. The authors suggest using K=2 or a smaller batch size as a compromise.
  • Performance Preservation vs. Improvement: Under strong augmentation settings, EnSiam primarily preserves performance rather than significantly improving it.
  • Future Work: The authors suggest that EnSiam's simple and flexible nature makes it extensible to other self-supervised learning methods to improve their performance. Combining EnSiam with specialized methods for strong augmentations could lead to even higher performance.

Summary updated

BibTeX
@article{han2024self,
  author = {Kyoungmin Han and Minsik Lee},
  title = {{Self-Supervised Learning with Ensemble Representations}},
  journal = {Engineering Applications of AI},
  year = {2024},
}
Figure