Unsupervised Sentence Representation via Contrastive Learning with Mixing Negatives

Yanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu, Yongyi Mao

Abstract

Unsupervised sentence representation learning has seen significant progress with contrastive methods like SimCSE. However, these methods often suffer from "gradient starvation" as training progresses because randomly sampled negative pairs become too easy for the model to distinguish. We present MixCSE, a novel approach that improves sentence embeddings by synthesizing "hard negatives" through the interpolation of positive and negative features. By injecting these artificial challenges, MixCSE maintains a strong gradient signal throughout training. Empirical results on Semantic Textual Similarity (STS) and downstream transfer tasks demonstrate that MixCSE consistently outperforms existing state-of-the-art unsupervised models.

Introduction

The goal of unsupervised sentence representation is to map sentences into a vector space where semantic similarity corresponds to spatial proximity. Contrastive learning achieves this by pulling positive pairs (e.g., different dropout masks of the same sentence) together and pushing negative pairs (randomly sampled sentences) apart. While effective, the authors of MixCSE identify a critical flaw: as the model learns to distribute embeddings more uniformly across the hypersphere, randomly sampled negatives become increasingly distinct from the anchor. This leads to a vanishing gradient, where the model stops receiving useful signals for further refinement, leaving the representations sub-optimal.

The MixCSE Approach

To counteract the "easy negative" trap, MixCSE generates "hard negatives" on the fly. Instead of searching for difficult samples in the dataset, which is computationally expensive, MixCSE creates them through feature mixing.

Conceptual illustration of MixCSE: The mixed negative is synthesized to be closer to the anchor than random negatives, creating a harder optimization challenge.

For an anchor sentence $h_i$, a "mix negative" $\tilde{h}_{i,j}$ is constructed by taking a weighted average of the anchor's positive feature $h'_i$ and a random negative feature $h'_j$: $$\tilde{h}_{i,j} = \text{norm}(\lambda h'_i + (1-\lambda)h'_j)$$ By incorporating a portion of the positive feature, the resulting vector is guaranteed to be closer to the anchor than a purely random negative, thereby providing a "harder" challenge for the contrastive loss.

Comparison between MixCSE and traditional contrastive methods. MixCSE introduces a mixing step to synthesize challenging negatives from existing features.

A key technical component is the Stop-Gradient (SG) operator applied to the mixed negative. This ensures that the model does not try to minimize the loss by moving the positive feature away from the anchor, but rather focuses solely on pushing the synthesized negative feature further away.

Theoretical Motivation

The paper provides a mathematical foundation for why hard negatives are necessary. In high-dimensional spaces, random vectors tend to be nearly orthogonal. As contrastive learning improves the uniformity of the embedding space, the probability of a random negative being close to the anchor decreases. The authors prove that without hard negatives, the gradient of the standard contrastive loss decreases toward zero as uniformity increases. MixCSE bypasses this by ensuring that challenging negatives are always present, regardless of the overall distribution's uniformity.

Experimental Results

MixCSE was evaluated using 1 million unlabeled sentences from Wikipedia for training.

  • STS Benchmarks: MixCSE-BERT$_{base}$ achieved an average score of 77.66, a significant improvement over SimCSE's 74.83. The large-scale version reached 78.80.
  • Transfer Tasks: On seven standard classification tasks (e.g., MR, SST-2, TREC), MixCSE-BERT$_{large}$ achieved a state-of-the-art average accuracy of 87.77% for unsupervised methods.
  • Robustness: The model showed lower variance across multiple runs compared to SimCSE, indicating that the synthesized negatives stabilize the optimization process.

Conclusion

MixCSE demonstrates that the quality of negative samples is paramount in contrastive learning for sentence embeddings. By synthesizing hard negatives through simple linear interpolation, the model avoids training stagnation and achieves superior alignment and uniformity. This approach provides a computationally efficient and theoretically grounded path toward more robust unsupervised representations.

Summary updated

Abstract

Unsupervised sentence representation learning is a fundamental problem in natural language processing. Recently, contrastive learning has made great success on this task. Existing constrastive learning based models usually apply random sampling to select negative examples for training. Previous work in computer vision has shown that hard negative examples help contrastive learning to achieve faster convergency and better optimization for representation learning. However, the importance of hard negatives in contrastive learning for sentence representation is yet to be explored. In this study, we prove that hard negatives are essential for maintaining strong gradient signals in the training process while random sampling negative examples is ineffective for sentence representation. Accordingly, we present a contrastive model, MixCSE, that extends the current state-of-the-art SimCSE by continually constructing hard negatives via mixing both positive and negative features. The superior performance of the proposed approach is demonstrated via empirical studies on Semantic Textual Similarity datasets and Transfer task datasets.
BibTeX
@inproceedings{zhang2022unsupervised,
  author = {Yanzhao Zhang and Richong Zhang and Samuel Mensah and Xudong Liu and Yongyi Mao},
  title = {{Unsupervised Sentence Representation via Contrastive Learning with Mixing Negatives}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21428},
}
Figure