Angle-optimized Text Embeddings

Xianming Li, Jing LiSubmission to ICLR 2024 · Withdrawn

TL;DR

Standard text embedding models suffer from vanishing gradients because cosine similarity plateaus near extreme values (its "saturation zones"), making it hard to differentiate subtly distinct texts. The authors introduce AnglE, a framework that maps sentence embeddings into a complex vector space and directly optimizes the angular difference between them. This approach prevents gradient death, improves both transfer and non-transfer Semantic Textual Similarity (STS) benchmarks, and scales effectively to long-form documents.

What Problem Does This Paper Address?

Most supervised sentence embedding models (such as Sentence-BERT) rely on cosine similarity in their loss functions. However, the derivative of $\cos(\theta)$ approaches zero when $\theta$ is near $0$ or $\pi$—regions known as saturation zones.

When two sentences are very similar (e.g., entailment vs. subtle neutral pairs in NLI datasets), their embeddings land squarely in these saturation zones. The resulting gradients during backpropagation vanish ($\Delta y \approx 0$), preventing the neural network from learning fine-grained semantic boundaries. This causes models to frequently misclassify near-duplicate or closely related sentences.

Methods

AnglE tackles gradient saturation by optimizing angles directly in the complex domain:

  • Complex Vector Representation: Instead of treating an embedding as a flat real vector of size $2D$, AnglE partitions it into real ($\mathbf{X}^{\text{re}} \in \mathbb{R}^D$) and imaginary ($\mathbf{X}^{\text{im}} \in \mathbb{R}^D$) components, forming a complex vector $z = \mathbf{a} + \mathbf{b}i$.
  • Complex Space Division for Angle Difference: Given two complex sentence vectors $z$ and $w$, their quotient in polar form isolates their angle difference: $$\frac{z}{w} = \left(\frac{r_z}{r_w}\right) e^{i(\theta_z - \theta_w)}$$ By normalizing by the magnitude ratio $\gamma = r_z / r_w$, AnglE extracts the absolute normalized angular difference $\Delta\theta_{zw}$ without relying on trigonometric functions that saturate.

  • Combined Objective Function: The final training loss balances three objectives: $$\mathcal{L} = w_1 \mathcal{L}_{\text{cos}} + w_2 \mathcal{L}_{\text{ibn}} + w_3 \mathcal{L}_{\text{angle}}$$ where $\mathcal{L}_{\text{cos}}$ preserves standard ranking, $\mathcal{L}_{\text{ibn}}$ leverages in-batch negatives (equipped with an Identical Sentence Pair filter to remove noise), and $\mathcal{L}_{\text{angle}}$ forces similar pairs to minimize their angular difference.

Datasets

  • Standard STS Benchmarks: Evaluated on STS 2012–2016, STS-B, SICK-R, MRPC, QQP, and QNLI (with models pretrained/transferred from NLI corpora like MNLI and SNLI).
  • GitHub Issues Similarity Dataset (New): A newly curated long-text benchmark of ~22K sentence/document pairs mined from 55 major open-source GitHub repositories. Duplicate issue threads act as positive pairs and non-duplicates as negatives. Over 60% of the texts exceed 512 tokens, filling a gap in long-sequence STS evaluation.
  • Image-Caption Retrieval: Flickr30k test set for top-$k$ text-to-text retrieval.

Key Insights and Ideas

  • Linearizing Angular Optimization: In complex coordinate space, angle differences remain mathematically distinct even when cosine differences compress toward zero. This supplies non-zero gradients in regions where traditional models stall.
  • True Distribution Matching: Plotting density curves of predicted similarities against ground-truth labels shows that standard models collapse into narrow clusters. AnglE's similarity distributions closely mirror the multimodal ground truth, especially at the extreme ends ($0\text{--}1$ and $4\text{--}5$ rating intervals).
  • LLM-Supervised Bootstrapping: For domains lacking paired training data, off-the-shelf LLMs (e.g., ChatGPT, LLaMA 2, ChatGLM) can generate synthetic positive (synonym) and negative (antonym) pairs. Training AnglE on these LLM-generated pairs consistently beats unsupervised contrastive baselines.

Key Results

  • Transfer STS Tasks: AnglE-BERT achieved an average Spearman correlation of 82.37% (surpassing SimCSE-BERT’s 81.57%). Paired with LLaMA 2-7B and LoRA, AnglE-LLaMA2-7B achieved 85.96%, an improvement over SimCSE-LLaMA2-7B (85.24%).
  • Non-Transfer STS Tasks: AnglE-BERT reached 73.55% average Spearman correlation across five benchmarks, outperforming standard SBERT (68.03%) and SimCSE-BERT (56.72%).
  • Long-Text STS: When paired with a long-sequence backbone (RAN$_{\text{base}}$), AnglE achieved 71.25% on the GitHub Issues dataset, outperforming the short-text BERT baseline (70.55%).
  • Text Retrieval: On Flickr30k caption matching, AnglE attained 12.9% strict top-5 retrieval accuracy, beating supervised SimCSE (10.4%) and SBERT (5.2%).

Limitations and Future Work

  • Hardware and Batch Size Constraints: Contrastive and in-batch objectives scale with larger mini-batches. Experiments were capped at a batch size of 50 due to 24 GB GPU memory limits (NVIDIA RTX 3090 Ti). Scaling to larger hardware (e.g., A100s) remains future work.
  • Supervision Reliance: While the authors demonstrate that synthetic data from LLMs offsets data scarcity, AnglE fundamentally relies on paired/labeled data rather than entirely unsupervised learning.

Summary updated

Abstract

High-quality text embedding is pivotal in improving semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. However, a common challenge existing text embedding models face is the problem of vanishing gradients, primarily due to their reliance on the cosine function in the optimization objective, which has saturation zones. To address this issue, this paper proposes a novel angle-optimized text embedding model called AnglE. The core idea of AnglE is to introduce angle optimization in a complex space. This novel approach effectively mitigates the adverse effects of the saturation zone in the cosine function, which can impede gradient and hinder optimization processes. To set up a comprehensive STS evaluation, we experimented on existing short-text STS datasets and a newly collected long-text STS dataset from GitHub Issues. Furthermore, we examine domain-specific STS scenarios with limited labeled data and explore how AnglE works with LLM-annotated data. Extensive experiments were conducted on various tasks including short-text STS, long-text STS, and domain-specific STS tasks. The results show that AnglE outperforms the state-of-the-art (SOTA) STS models that ignore the cosine saturation zone. These findings demonstrate the ability of AnglE to generate high-quality text embeddings and the usefulness of angle optimization in STS.
BibTeX
@misc{li2023angle,
  author = {Xianming LI and Jing Li},
  title = {{Angle-optimized Text Embeddings}},
  year = {2023},
  eprint = {2309.12871},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.12871},
}

Loading related content...

Figure