TL;DR
This paper introduces RAOE, a novel framework for learning sentence embeddings by addressing the inherent conflict between discrete Natural Language Inference (NLI) labels and continuous semantic similarity. RAOE leverages both NLI labels and pre-computed similarity scores with a composite objective, achieving state-of-the-art performance on Semantic Textual Similarity (STS) tasks and strong generalization across various NLP benchmarks.
What Problem Does This Paper Address?
Learning high-quality sentence embeddings is crucial for many NLP tasks, and Natural Language Inference (NLI) datasets like SNLI and MNLI have been a popular source for supervised training. However, a significant challenge arises from the mismatch between the discrete NLI labels (Entailment, Neutral, Contradiction) and the continuous nature of semantic similarity. As illustrated in Figure 1, there's substantial overlap between the similarity distributions for different NLI labels, meaning discrete labels don't perfectly capture nuanced semantic relationships. Traditional methods often discard "Neutral" pairs, losing valuable information, or make strong assumptions about label ordering that are contradicted by this overlap. This leads to suboptimal performance, especially on fine-grained tasks like STS.
Methods
RAOE (Rank-Awareness and Angular Optimization Embeddings) tackles this by introducing a novel composite objective that integrates continuous similarity scores (S) with discrete NLI labels (L).
- Enhanced NLI Data Strategy: RAOE uses the full NLI dataset (Entailment, Neutral, Contradiction) and augments each sentence pair with a pre-computed continuous similarity score (S). This score is derived from the average cosine similarity of two high-performance embedding models, providing a richer supervisory signal.
- Composite Objective: The core of RAOE is its dual-objective function:
- Rank Margin Objective (Lrank): This component enforces consistency between the ranking of sentence pairs based on the pre-computed similarity scores (S) and the ranking based on learned cosine similarities. It uses an explicit rank margin to focus on pairs with significant similarity differences, promoting robust rank ordering.
- Gated Angular Objective (Langle): This objective refines embedding geometry by considering the angles between embeddings. It employs a gating mechanism that activates optimization only when the NLI label (L) and the continuous similarity score (S) agree on the relative semantic relatedness of a pair. This conditional approach helps mitigate conflicting signals and improve embedding space structure.
Datasets
- Training: SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) datasets were used for training.
- Evaluation:
- STS Benchmarks: STS12-STS16, STS Benchmark (STSb), and SICK-Relatedness (SICK-R).
- Generalization: SentEval transfer learning suite and the Massive Text Embedding Benchmark (MTEB), which comprises 56 diverse tasks.
Key Insights and Ideas
- Discrete labels are insufficient: The significant overlap in similarity scores across NLI labels (Figure 1) is a critical insight, highlighting why relying solely on discrete labels is problematic.
- Continuous scores are vital: Augmenting NLI data with continuous similarity scores provides a much richer and more coherent supervisory signal.
- Synergy of Rank and Angle: The combination of rank-awareness (Lrank) and conditional angular optimization (Langle) is more effective than either component alone, leading to more robust and nuanced representations.
- Leveraging Neutral pairs: RAOE demonstrates that Neutral pairs, often discarded, can be effectively utilized when guided by continuous similarity scores, contributing positively to performance.
- Task-specific specialization: The optional Stage 2 fine-tuning (RAOE-S2) shows that specializing embeddings for STS tasks can yield further significant improvements, though it may reduce generalization.
Key Results
- STS Benchmark Performance: RAOE-S1 (BERT-base) achieved an average Spearman correlation of 85.11, outperforming SimCSE (81.57) and AnglE (82.43).
- State-of-the-Art STS: The STS-specialized RAOE-S2 model achieved a new state-of-the-art performance on STS tasks, with BERT-large reaching an average correlation of 88.90 and Qwen2.5-7B reaching 89.37.
- Generalization (SentEval & MTEB): RAOE-S1 consistently outperformed strong baselines like SimCSE on SentEval transfer tasks (e.g., 87.85 average accuracy for BERT-base vs. 85.81 for SimCSE) and MTEB benchmarks (e.g., 56.50 average score for ModernBERT-large vs. 53.52 for SimCSE).
- Efficiency: RAOE is computationally efficient, matching the efficiency of InfoNCE while delivering superior accuracy.
Limitations and Future Work
- Reliance on pre-computed scores: RAOE's performance is dependent on the quality of the pre-computed similarity scores (S), potentially inheriting limitations from the source embedding models.
- Geometric analysis: The precise geometric effects of the composite objective in high-dimensional spaces warrant further investigation.
- Specialization vs. Generalization: The STS-specialized RAOE-S2 model shows reduced generalization compared to RAOE-S1, indicating a trade-off for specific applications.
- Future Work: The authors suggest exploring dynamic similarity score generation, deeper geometric analysis, and extending RAOE to diverse languages or domains.
Summary updated