TL;DR
Standard text embedding models suffer from vanishing gradients because cosine similarity plateaus near extreme values (its "saturation zones"), making it hard to differentiate subtly distinct texts. The authors introduce AnglE, a framework that maps sentence embeddings into a complex vector space and directly optimizes the angular difference between them. This approach prevents gradient death, improves both transfer and non-transfer Semantic Textual Similarity (STS) benchmarks, and scales effectively to long-form documents.
What Problem Does This Paper Address?
Most supervised sentence embedding models (such as Sentence-BERT) rely on cosine similarity in their loss functions. However, the derivative of $\cos(\theta)$ approaches zero when $\theta$ is near $0$ or $\pi$—regions known as saturation zones.
When two sentences are very similar (e.g., entailment vs. subtle neutral pairs in NLI datasets), their embeddings land squarely in these saturation zones. The resulting gradients during backpropagation vanish ($\Delta y \approx 0$), preventing the neural network from learning fine-grained semantic boundaries. This causes models to frequently misclassify near-duplicate or closely related sentences.
Methods
AnglE tackles gradient saturation by optimizing angles directly in the complex domain:
- Complex Vector Representation: Instead of treating an embedding as a flat real vector of size $2D$, AnglE partitions it into real ($\mathbf{X}^{\text{re}} \in \mathbb{R}^D$) and imaginary ($\mathbf{X}^{\text{im}} \in \mathbb{R}^D$) components, forming a complex vector $z = \mathbf{a} + \mathbf{b}i$.
-
Complex Space Division for Angle Difference: Given two complex sentence vectors $z$ and $w$, their quotient in polar form isolates their angle difference: $$\frac{z}{w} = \left(\frac{r_z}{r_w}\right) e^{i(\theta_z - \theta_w)}$$ By normalizing by the magnitude ratio $\gamma = r_z / r_w$, AnglE extracts the absolute normalized angular difference $\Delta\theta_{zw}$ without relying on trigonometric functions that saturate.
-
Combined Objective Function: The final training loss balances three objectives: $$\mathcal{L} = w_1 \mathcal{L}_{\text{cos}} + w_2 \mathcal{L}_{\text{ibn}} + w_3 \mathcal{L}_{\text{angle}}$$ where $\mathcal{L}_{\text{cos}}$ preserves standard ranking, $\mathcal{L}_{\text{ibn}}$ leverages in-batch negatives (equipped with an Identical Sentence Pair filter to remove noise), and $\mathcal{L}_{\text{angle}}$ forces similar pairs to minimize their angular difference.
Datasets
- Standard STS Benchmarks: Evaluated on STS 2012–2016, STS-B, SICK-R, MRPC, QQP, and QNLI (with models pretrained/transferred from NLI corpora like MNLI and SNLI).
- GitHub Issues Similarity Dataset (New): A newly curated long-text benchmark of ~22K sentence/document pairs mined from 55 major open-source GitHub repositories. Duplicate issue threads act as positive pairs and non-duplicates as negatives. Over 60% of the texts exceed 512 tokens, filling a gap in long-sequence STS evaluation.
- Image-Caption Retrieval: Flickr30k test set for top-$k$ text-to-text retrieval.
Key Insights and Ideas
- Linearizing Angular Optimization: In complex coordinate space, angle differences remain mathematically distinct even when cosine differences compress toward zero. This supplies non-zero gradients in regions where traditional models stall.
- True Distribution Matching: Plotting density curves of predicted similarities against ground-truth labels shows that standard models collapse into narrow clusters. AnglE's similarity distributions closely mirror the multimodal ground truth, especially at the extreme ends ($0\text{--}1$ and $4\text{--}5$ rating intervals).
- LLM-Supervised Bootstrapping: For domains lacking paired training data, off-the-shelf LLMs (e.g., ChatGPT, LLaMA 2, ChatGLM) can generate synthetic positive (synonym) and negative (antonym) pairs. Training AnglE on these LLM-generated pairs consistently beats unsupervised contrastive baselines.
Key Results
- Transfer STS Tasks: AnglE-BERT achieved an average Spearman correlation of 82.37% (surpassing SimCSE-BERT’s 81.57%). Paired with LLaMA 2-7B and LoRA, AnglE-LLaMA2-7B achieved 85.96%, an improvement over SimCSE-LLaMA2-7B (85.24%).
- Non-Transfer STS Tasks: AnglE-BERT reached 73.55% average Spearman correlation across five benchmarks, outperforming standard SBERT (68.03%) and SimCSE-BERT (56.72%).
- Long-Text STS: When paired with a long-sequence backbone (RAN$_{\text{base}}$), AnglE achieved 71.25% on the GitHub Issues dataset, outperforming the short-text BERT baseline (70.55%).
- Text Retrieval: On Flickr30k caption matching, AnglE attained 12.9% strict top-5 retrieval accuracy, beating supervised SimCSE (10.4%) and SBERT (5.2%).
Limitations and Future Work
- Hardware and Batch Size Constraints: Contrastive and in-batch objectives scale with larger mini-batches. Experiments were capped at a batch size of 50 due to 24 GB GPU memory limits (NVIDIA RTX 3090 Ti). Scaling to larger hardware (e.g., A100s) remains future work.
- Supervision Reliance: While the authors demonstrate that synthetic data from LLMs offsets data scarcity, AnglE fundamentally relies on paired/labeled data rather than entirely unsupervised learning.
Summary updated