AoE: Angle-optimized Embeddings for Semantic Textual Similarity

Xianming Li, Jing LiACL 2024 · OpenReview

Abstract Text embedding is pivotal in semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. STS learning largely relies on the cosine function as the optimization objective to reflect semantic similarity. However, the cosine has saturation zones rendering vanishing gradients and hindering learning subtle semantic differences in text embeddings. To address this issue, we propose a novel Angle-optimized Embedding model, AoE. It optimizes angle differences in complex space to explore similarity in saturation zones better. To set up a comprehensive evaluation, we experimented with existing short-text STS, our newly collected long-text STS, and downstream task datasets. Extensive experimental results on STS and MTEB benchmarks show that AoE significantly outperforms popular text embedding models neglecting cosine saturation zones. It highlights that AoE can produce high-quality text embeddings and broadly benefit downstream tasks.

Introduction: The Cosine Saturation Problem

Most modern text embedding models, such as SBERT or SimCSE, rely on cosine similarity as their primary optimization objective. While effective for general similarity, the cosine function suffers from a significant mathematical limitation: saturation zones. As two text snippets become increasingly similar in surface appearance (sharing many tokens), their embeddings move into a region where the gradient of the cosine function approaches zero.

The saturation zone problem in cosine similarity leads to vanishing gradients, making it difficult for models to distinguish between entailment and neutral relationships in highly similar sentence pairs.

This "gradient vanishing" effect prevents the model from learning subtle semantic nuances. For instance, a model might incorrectly predict "entailment" for a pair of sentences that are actually "neutral" simply because they share 90% of their vocabulary. In these saturated regions, the model lacks the signal needed to fine-tune the embedding space further.

Methodology: Angle-optimized Embeddings (AoE)

To overcome these limitations, the researchers developed AoE (Angle-optimized Embeddings). This framework shifts the representation of text from real-valued vectors to complex space.

The AoE framework architecture, showing the transition from token embeddings to complex text embeddings and the subsequent angle optimization pipeline.

The AoE approach consists of four key components:

  1. Complex Decomposition: Instead of a single vector, AoE generates a real component ($X^{re}$) and an imaginary component ($X^{im}$). The real part typically captures surface-level "appearance" similarity, while the imaginary part is designed to capture subtle semantic differences.
  2. Complex Division: To compare two texts, the model performs complex division. This operation naturally yields an angle difference ($\Delta\theta$) and a magnitude factor, providing a more robust metric for comparison than simple dot products.
  3. Angle Optimization: Instead of optimizing the cosine of the angle, AoE optimizes the angle difference itself. By linearizing the objective, the model maintains a healthy gradient even in regions where vectors are nearly aligned, allowing for precise fine-tuning in what were previously "saturated" zones.
  4. Multi-Objective Learning: The training loss combines this new angle objective with a standard supervised contrastive loss. This dual approach allows the model to learn from both traditional similarity metrics and the new, more granular angle-based perspective.

Experimental Setup and Datasets

AoE was evaluated across a diverse set of benchmarks to ensure its robustness:

  • Standard Short-Text STS: Including STS 2012–2016, STS-B, and SICK-R.
  • MTEB (Massive Text Embedding Benchmark): A comprehensive suite of 56 datasets covering classification, clustering, and retrieval tasks.
  • GIS (GitHub Issues Similarity): A new dataset introduced by the authors containing ~22k samples of duplicate and non-duplicate GitHub issues. This dataset specifically targets long-form text (often exceeding 512 tokens) and technical content, which is traditionally challenging for embedding models.

Key Results and Insights

The experimental results demonstrate that AoE consistently outperforms existing state-of-the-art models:

  • Superior Accuracy: AoE-LLaMA-13B achieved a global best average score of 86.52 on standard STS tasks, surpassing popular models like SimCSE and SBERT.
  • Outperforming Closed-Source Models: AoE demonstrated higher performance than OpenAI’s text-embedding-3 and ada-002 on several key STS benchmarks.
  • MTEB Leadership: AoE achieved State-of-the-Art (SOTA) results on the MTEB leaderboard for BERT-large scale models, outperforming established models such as BGE and Ember.
  • Long-Text Robustness: On the GIS dataset, AoE showed a significant improvement over SimCSE (70.59 vs 60.38 Spearman correlation), highlighting its effectiveness for complex, technical, and long-form content.

Visualization via t-SNE further confirmed that the imaginary embeddings are more "scattered" than the real ones, indicating that the imaginary space successfully captures the nuances that surface-level representations ignore. Furthermore, AoE is architecture-agnostic, showing improvements across various backbones from BERT to LLaMA, with negligible increases in training time.

Conclusion

AoE represents a significant step forward in text embedding by addressing the fundamental mathematical flaw of cosine saturation. By leveraging complex space and direct angle optimization, it enables models to distinguish subtle semantic differences that were previously invisible. While challenges remain in long-text accuracy, AoE provides a powerful, efficient, and universally applicable framework for improving semantic textual similarity across a wide range of NLP applications.

Summary updated

Abstract

Text embedding is pivotal in semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. STS learning largely relies on the cosine function as the optimization objective to reflect semantic similarity. However, the cosine has saturation zones rendering vanishing gradients and hindering learning subtle semantic differences in text embeddings. To address this issue, we propose a novel Angle-optimized Embedding model, AoE. It optimizes angle differences in complex space to explore similarity in saturation zones better. To set up a comprehensive evaluation, we experimented with existing short-text STS, our newly collected long-text STS, and downstream task datasets. Extensive experimental results on STS and MTEB benchmarks show that AoE significantly outperforms popular text embedding models neglecting cosine saturation zones. It highlights that AoE can produce high-quality text embeddings and broadly benefit downstream tasks.
BibTeX
@inproceedings{li2024aoe,
  author = {Xianming Li and Jing Li},
  title = {{AoE: Angle-optimized Embeddings for Semantic Textual Similarity}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=2LwFogLjbC},
}
Figure