Abstract
Unsupervised sentence representation learning has seen significant progress with contrastive methods like SimCSE. However, these methods often suffer from "gradient starvation" as training progresses because randomly sampled negative pairs become too easy for the model to distinguish. We present MixCSE, a novel approach that improves sentence embeddings by synthesizing "hard negatives" through the interpolation of positive and negative features. By injecting these artificial challenges, MixCSE maintains a strong gradient signal throughout training. Empirical results on Semantic Textual Similarity (STS) and downstream transfer tasks demonstrate that MixCSE consistently outperforms existing state-of-the-art unsupervised models.
Introduction
The goal of unsupervised sentence representation is to map sentences into a vector space where semantic similarity corresponds to spatial proximity. Contrastive learning achieves this by pulling positive pairs (e.g., different dropout masks of the same sentence) together and pushing negative pairs (randomly sampled sentences) apart. While effective, the authors of MixCSE identify a critical flaw: as the model learns to distribute embeddings more uniformly across the hypersphere, randomly sampled negatives become increasingly distinct from the anchor. This leads to a vanishing gradient, where the model stops receiving useful signals for further refinement, leaving the representations sub-optimal.
The MixCSE Approach
To counteract the "easy negative" trap, MixCSE generates "hard negatives" on the fly. Instead of searching for difficult samples in the dataset, which is computationally expensive, MixCSE creates them through feature mixing.
For an anchor sentence $h_i$, a "mix negative" $\tilde{h}_{i,j}$ is constructed by taking a weighted average of the anchor's positive feature $h'_i$ and a random negative feature $h'_j$: $$\tilde{h}_{i,j} = \text{norm}(\lambda h'_i + (1-\lambda)h'_j)$$ By incorporating a portion of the positive feature, the resulting vector is guaranteed to be closer to the anchor than a purely random negative, thereby providing a "harder" challenge for the contrastive loss.
A key technical component is the Stop-Gradient (SG) operator applied to the mixed negative. This ensures that the model does not try to minimize the loss by moving the positive feature away from the anchor, but rather focuses solely on pushing the synthesized negative feature further away.
Theoretical Motivation
The paper provides a mathematical foundation for why hard negatives are necessary. In high-dimensional spaces, random vectors tend to be nearly orthogonal. As contrastive learning improves the uniformity of the embedding space, the probability of a random negative being close to the anchor decreases. The authors prove that without hard negatives, the gradient of the standard contrastive loss decreases toward zero as uniformity increases. MixCSE bypasses this by ensuring that challenging negatives are always present, regardless of the overall distribution's uniformity.
Experimental Results
MixCSE was evaluated using 1 million unlabeled sentences from Wikipedia for training.
- STS Benchmarks: MixCSE-BERT$_{base}$ achieved an average score of 77.66, a significant improvement over SimCSE's 74.83. The large-scale version reached 78.80.
- Transfer Tasks: On seven standard classification tasks (e.g., MR, SST-2, TREC), MixCSE-BERT$_{large}$ achieved a state-of-the-art average accuracy of 87.77% for unsupervised methods.
- Robustness: The model showed lower variance across multiple runs compared to SimCSE, indicating that the synthesized negatives stabilize the optimization process.
Conclusion
MixCSE demonstrates that the quality of negative samples is paramount in contrastive learning for sentence embeddings. By synthesizing hard negatives through simple linear interpolation, the model avoids training stagnation and achieves superior alignment and uniformity. This approach provides a computationally efficient and theoretically grounded path toward more robust unsupervised representations.
Summary updated