TL;DR
This paper explores non-contrastive self-supervised learning methods for sentence embeddings, specifically comparing Barlow Twins and VICReg against the popular SimCSE. They find that dimension-contrastive methods like Barlow Twins can match or even outperform SimCSE on downstream tasks, challenging the dominance of contrastive approaches in this domain.
What Problem Does This Paper Address?
Learning effective sentence embeddings is crucial for a wide range of Natural Language Processing (NLP) tasks, from document clustering to information retrieval. Traditionally, many state-of-the-art unsupervised methods rely on contrastive learning. These methods require identifying "positive" pairs (similar sentences) and "negative" pairs (dissimilar sentences) to train a model. This often involves complex data augmentation strategies or curated datasets.
This paper investigates an alternative: non-contrastive self-supervised learning, specifically "dimension-contrastive" methods. These methods aim to learn good representations by regularizing the embedding space itself, rather than explicitly contrasting positive and negative examples. The authors want to see if these dimension-contrastive approaches, which have shown promise in computer vision, can be competitive with established contrastive methods like SimCSE for sentence embeddings.
Methods
The core of this work is a comparative study of different self-supervised learning objectives for sentence embeddings.
- SimCSE (Sample Contrastive): This is the authors' baseline, representing the standard approach. Unsupervised SimCSE uses different dropout masks on the same input sentence to create two augmented "views" and trains the model to make these views similar using a contrastive loss.
- Barlow Twins (BT) (Dimension Contrastive): This method aims to make the cross-correlation matrix between the embeddings of two augmented views of a sentence close to the identity matrix. It does this by penalizing the difference between the cross-correlation matrix and the identity matrix, and also by decorrelating the different dimensions of the embeddings.
- VICReg (Dimension Contrastive): Similar to Barlow Twins, VICReg also uses dimension-wise regularization. Its loss function has three terms: one to make the embeddings of augmented views similar, one to encourage variance of embedding dimensions to be around 1 (preventing collapse), and one to decorrelate embedding dimensions.
The authors also experiment with different data augmentation strategies for BT and VICReg, including:
- Dropout: Similar to SimCSE, using different dropout masks.
- Shuffling: Randomly permuting a fraction of input tokens.
- EDA (Easy Data Augmentation): A set of text augmentation techniques including synonym replacement, random insertion, random swap, and random deletion.
Datasets
- Training: The primary dataset used for unsupervised training is Wikipedia. For supervised experiments, they use subsets of SNLI and MNLI (for entailment pairs) and WikiAuto (aligned English Wikipedia sentences).
- Evaluation: The main evaluation benchmark is the Massive Text Embedding Benchmark (MTEB), which comprises 56 English language datasets covering tasks like classification, clustering, retrieval, and semantic textual similarity (STS). They also use the STS-B dev set for hyperparameter tuning and checkpoint selection.
Key Insights and Ideas
- Dimension Contrastive Methods are Competitive: The most significant finding is that dimension-contrastive methods like Barlow Twins can achieve performance comparable to, or even exceeding, unsupervised SimCSE on various downstream tasks. This suggests that explicit positive/negative pair mining might not be strictly necessary for strong sentence embeddings.
- Barlow Twins Outperforms VICReg: Across experiments, Barlow Twins consistently performed better than VICReg, which was found to be harder to optimize.
- Augmentation Matters: The choice of data augmentation significantly impacts performance. For Barlow Twins and VICReg, shuffling proved to be a strong augmentation strategy, often outperforming dropout. EDA, however, generally underperformed.
- Supervision Doesn't Always Help: While supervised methods (using NLI or WikiAuto) can outperform unsupervised ones on specific tasks, there isn't a clear universal winner. This aligns with the observation that no single model consistently dominates across all tasks.
- Alignment vs. Uniformity Trade-off: The paper touches upon the alignment and uniformity metrics, noting that while non-contrastive methods can improve uniformity, they might degrade alignment. However, this doesn't necessarily translate to worse downstream performance, suggesting a complex relationship.
Key Results
- Barlow Twins vs. SimCSE: On the MTEB benchmark, RoBERTa models trained with Barlow Twins and dropout augmentation outperformed SimCSE on most tasks. Barlow Twins with shuffle augmentation also showed strong results.
- VICReg Performance: VICReg generally underperformed both SimCSE and Barlow Twins, with the authors attributing this to optimization challenges.
- Augmentation Impact: For Barlow Twins, shuffle augmentation with a high fraction (0.5) yielded some of the best results, outperforming dropout in many cases. EDA was consistently the weakest augmentation.
- Model Performance: RoBERTa-based models generally outperformed BERT-based models across the board.
- STS-B Overfitting: The authors note that VICReg's strong performance on the STS-B dev set (used for checkpoint selection) might indicate overfitting, as this didn't always translate to better performance on other downstream tasks.
Limitations and Future Work
- Limited Scope of Non-Contrastive Methods: The study focused on Barlow Twins and VICReg. The authors suggest exploring other dimension-contrastive methods.
- Baseline Simplicity: While SimCSE is a strong baseline, future work could explore other sample-contrastive methods like DiffCSE or InfoCSE.
- Model Architectures: The study was limited to BERT and RoBERTa base models. Exploring decoder-only or encoder-decoder architectures would be beneficial.
- Multilingual Capabilities: The current study is limited to English documents. Extending these methods to multilingual corpora is a promising avenue.
- Optimization of VICReg: The authors acknowledge that VICReg's underperformance might be due to insufficient hyperparameter tuning and suggest further exploration.
Summary updated