Improving Contrastive Learning of Sentence Embeddings from AI Feedback

Qinyuan Cheng, Xiaogui Yang, Tianxiang Sun, Linyang Li, Xipeng Qiu

Abstract: Contrastive learning has become a popular approach in natural language processing, particularly for the learning of sentence embeddings. However, the discrete nature of natural language makes it difficult to ensure the quality of positive and negative sample pairs generated through data augmentation methods. Although supervised contrastive learning can produce more accurate sample pairs with human feedback labels, it still lacks fine-grained training signals. In this paper, we propose to improve Contrastive Learning of sentence embeddings from AI Feedback (CLAIF). Our method utilizes AI feedback from large pre-trained language models (LLMs) to construct sample pairs with fine-grained sample similarity scores to improve contrastive learning. Besides, we combine human feedback and AI feedback to provide better supervision signals for supervised contrastive learning of sentence embeddings. Experimental results show that our method achieves state-of-the-art performance on several semantic textual similarity (STS) and transfer learning tasks compared to other unsupervised and supervised contrastive learning methods.

Introduction

Sentence embeddings are fundamental to modern information retrieval, semantic search, and document clustering. Despite the success of pre-trained models like BERT, they often suffer from "anisotropy," where embeddings are clustered in a narrow cone, leading to poor semantic differentiation. Contrastive learning aims to solve this by pulling similar sentences together and pushing dissimilar ones apart.

However, existing methods face significant hurdles. Unsupervised methods rely on noisy data augmentation (like dropout or word deletion), while supervised methods typically rely on human-labeled datasets like Natural Language Inference (NLI). These human labels are often coarse-grained (binary 0 or 1), failing to capture the nuanced spectrum of semantic similarity. To address these limitations, we introduce CLAIF, a framework that leverages the linguistic intelligence of Large Language Models (LLMs) to provide high-quality, fine-grained supervision signals.

The CLAIF Framework

Our approach involves a two-step pipeline designed to generate synthetic training data that captures a wide range of semantic relationships.

Performance comparison on SentEval benchmarks showing CLAIF variants achieving state-of-the-art results across BERT and RoBERTa architectures.

1. Sentence Pair Generation

To create diverse pairs, we take an original sentence and apply varying "mask rates" (from 0% to 80%). We then prompt GPT-3 to reconstruct the sentence based on the remaining fragments. This process naturally produces a spectrum of pairs: low mask rates yield near-duplicates, while high mask rates result in sentences that may only share a broad theme or be entirely different.

2. Semantic Similarity Labeling

Instead of assigning binary labels, we prompt the LLM to provide a continuous similarity score between 0.0 and 1.0 for each generated pair. This allows the model to learn from nuanced feedback—for example, recognizing that two sentences are 80% similar rather than just "similar."

Training Methodology

We propose two primary training variants:

  • CLAIF (AI Feedback): The model is trained using a Mean Squared Error (MSE) loss to directly predict the continuous similarity scores provided by the AI.
  • CLHAIF (Human and AI Feedback): In supervised settings, we use AI to refine existing human labels. By applying a Soft InfoNCE loss, the AI's continuous scores replace rigid one-hot labels, allowing the model to benefit from both human accuracy and AI granularity.

Experimental Results

We evaluated our models on 7 Semantic Textual Similarity (STS) tasks and 7 transfer learning tasks. The results demonstrate that AI feedback provides a significantly stronger supervision signal than traditional methods.

Performance comparison on SentEval benchmarks showing CLAIF variants achieving state-of-the-art results across BERT and RoBERTa architectures.
Note: While the figure above illustrates the pipeline, the following table summarizes the quantitative gains.
SentEval average scores for various models, demonstrating the superiority of CLAIF and CLHAIF methods.

Key findings include:

  • State-of-the-Art Performance: CLAIF-BERT achieved an average score of 79.63 on STS tasks, outperforming the widely used SimCSE (76.25).
  • Scalability: The CLAIF-scaled model, trained on over 1.2 million pairs, reached an average of 82.37, surpassing OpenAI’s text-ada-embedding-002 (80.82).
  • Cost Efficiency: Generating the scaled dataset cost approximately $720, a fraction of the cost required for human annotation of a similar volume of data.

Conclusion

CLAIF demonstrates that LLMs can serve as highly effective "teachers" for sentence embedding models. By providing fine-grained, continuous similarity scores, AI feedback overcomes the limitations of coarse human labels and noisy unsupervised augmentations. This approach is not only more accurate but also highly scalable, offering a new path for developing high-performance semantic representations without the prohibitive cost of large-scale human labeling.

Summary updated

Abstract

Contrastive learning has become a popular approach in natural language processing, particularly for the learning of sentence embeddings. However, the discrete nature of natural language makes it difficult to ensure the quality of positive and negative sample pairs generated through data augmentation methods. Although supervised contrastive learning can produce more accurate sample pairs with human feedback labels, it still lacks fine-grained training signals. In this paper, we propose to improve \textbf{C}ontrastive \textbf{L}earning of sentence embeddings from \textbf{AI} \textbf{F}eedback \textbf{(CLAIF)}. Our method utilizes AI feedback from large pre-trained language models (LLMs) to construct sample pairs with fine-grained sample similarity scores to improve contrastive learning. Besides, we combine human feedback and AI feedback to provide better supervision signals for supervised contrastive learning of sentence embeddings. Experimental results show that our method achieves state-of-the-art performance on several semantic textual similarity (STS) and transfer learning tasks compared to other unsupervised and supervised contrastive learning methods.
BibTeX
@inproceedings{cheng2023improving,
  author = {Qinyuan Cheng and Xiaogui Yang and Tianxiang Sun and Linyang Li and Xipeng Qiu},
  title = {{Improving Contrastive Learning of Sentence Embeddings from AI Feedback}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.707},
}
Figure