AnglE-optimized Text Embeddings

Xianming Li, Jing Li

Abstract

High-quality text embedding is pivotal in improving semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. However, a common challenge existing text embedding models face is the problem of vanishing gradients, primarily due to their reliance on the cosine function in the optimization objective, which has saturation zones. To address this issue, this paper proposes a novel angle-optimized text embedding model called AnglE. The core idea of AnglE is to introduce angle optimization in a complex space. This novel approach effectively mitigates the adverse effects of the saturation zone in the cosine function, which can impede gradient and hinder optimization processes. To set up a comprehensive STS evaluation, we experimented on existing short-text STS datasets and a newly collected long-text STS dataset from GitHub Issues. Furthermore, we examine domain-specific STS scenarios with limited labeled data and explore how AnglE works with LLM-annotated data. Extensive experiments were conducted on various tasks including short-text STS, long-text STS, and domain-specific STS tasks. The results show that AnglE outperforms the state-of-the-art (SOTA) STS models that ignore the cosine saturation zone. These findings demonstrate the ability of AnglE to generate high-quality text embeddings and the usefulness of angle optimization in STS.

1. Introduction

Semantic Textual Similarity (STS) is a fundamental task in natural language processing, serving as a backbone for applications ranging from information retrieval to retrieval-augmented generation (RAG) in Large Language Models (LLMs). Current state-of-the-art models, such as SBERT and SimCSE, typically optimize for cosine similarity between vector representations.

However, the cosine function possesses inherent "saturation zones"—regions near 1 and -1 where the function's curve becomes nearly flat. In these zones, the gradient approaches zero, leading to the "vanishing gradient" problem. This phenomenon prevents models from effectively learning fine-grained semantic distinctions, particularly when dealing with highly similar pairs or binary labels.

The cosine function exhibits saturation zones near 1 and -1, where gradients vanish.

2. Methodology: The AnglE Framework

To overcome the limitations of cosine-based optimization, we propose AnglE (Angle-optimized Text Embedding). The fundamental shift is moving optimization from a real-valued vector space to a complex space.

2.1 Complex Space Mapping

AnglE transforms standard embeddings into complex representations. Given an embedding vector, it is partitioned into real ($X^{re}$) and imaginary ($X^{im}$) components. This allows the model to represent text as complex numbers $z = a + bi$.

2.2 Angle Optimization

In complex space, the relationship between two vectors can be defined by their angle difference ($\Delta\theta$). Unlike the cosine function, which flattens out as vectors align, the angle difference remains distinct and provides a consistent gradient signal even for highly similar vectors.

AnglE optimizes the angle difference in complex space (a) to provide robust gradients where cosine similarity (b) saturates.

2.3 Objective Function

The AnglE training objective is a multi-faceted loss function:

  • Cosine Objective ($L_{cos}$): Maintains standard similarity alignment.
  • In-batch Negative Objective ($L_{ibn}$): A contrastive loss component using batch negatives to improve discriminative power.
  • Angle Objective ($L_{angle}$): The primary innovation that minimizes the normalized angle difference for similar pairs, ensuring robust optimization across all similarity levels.

3. Experimental Setup

We evaluated AnglE across a diverse set of benchmarks:

  • Standard STS: STS 2012-2016, STS-B, and SICK-R.
  • Transfer Tasks: MRPC, QQP, and QNLI.
  • GitHub Issues Similarity Dataset: A new benchmark introduced in this work consisting of ~21,000 samples from 55 open-source projects. This dataset specifically targets long-text STS, featuring extensive descriptions and code blocks that often exceed the 512-token limit of standard BERT models.

4. Results and Discussion

4.1 State-of-the-Art Performance

AnglE-LLaMA2-7B achieved a new SOTA average Spearman correlation of 85.96% on standard STS tasks, significantly outperforming previous benchmarks like SimCSE. In domain-specific, non-transfer settings, AnglE demonstrated a 5.52% absolute improvement over SBERT.

4.2 Long-Text and Retrieval Capabilities

On the GitHub Issues dataset, AnglE showed superior performance in handling long-form content. Furthermore, in retrieval tasks like Flickr30k, AnglE achieved a strict accuracy of 12.9%, nearly doubling the performance of SBERT (5.2%).

4.3 LLM-Supervised Training

We also explored the utility of LLMs as annotators. Our results indicate that AnglE can be effectively fine-tuned using data labeled by models like ChatGPT, offering a scalable solution for domain-specific applications where human-labeled data is scarce.

5. Conclusion

AnglE addresses a critical bottleneck in text embedding optimization by replacing or augmenting the saturated cosine objective with angle optimization in complex space. By providing more informative gradients, AnglE enables the learning of more nuanced semantic representations. Our extensive evaluations confirm that AnglE is highly effective for short-text, long-text, and domain-specific STS tasks, setting a new standard for high-quality text embeddings.

Code Availability: https://github.com/SeanLee97/AnglE

Summary updated

Abstract

High-quality text embedding is pivotal in improving semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. However, a common challenge existing text embedding models face is the problem of vanishing gradients, primarily due to their reliance on the cosine function in the optimization objective, which has saturation zones. To address this issue, this paper proposes a novel angle-optimized text embedding model called AnglE. The core idea of AnglE is to introduce angle optimization in a complex space. This novel approach effectively mitigates the adverse effects of the saturation zone in the cosine function, which can impede gradient and hinder optimization processes. To set up a comprehensive STS evaluation, we experimented on existing short-text STS datasets and a newly collected long-text STS dataset from GitHub Issues. Furthermore, we examine domain-specific STS scenarios with limited labeled data and explore how AnglE works with LLM-annotated data. Extensive experiments were conducted on various tasks including short-text STS, long-text STS, and domain-specific STS tasks. The results show that AnglE outperforms the state-of-the-art (SOTA) STS models that ignore the cosine saturation zone. These findings demonstrate the ability of AnglE to generate high-quality text embeddings and the usefulness of angle optimization in STS.
BibTeX
@misc{li2023angle,
  author = {Xianming Li and Jing Li},
  title = {{AnglE-optimized Text Embeddings}},
  year = {2023},
  eprint = {2309.12871},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.12871},
}
Figure