On Isotropy and Learning Dynamics of Contrastive-based Sentence Representation Learning

Chenghao Xiao, Yang Long, Noura Al Moubayed

Introduction

Contrastive learning (CL) has emerged as the definitive methodology for training high-performance sentence embeddings. Models such as SimCSE and MPNet consistently dominate benchmarks like the Semantic Textual Similarity (STS) tasks. However, the underlying geometric transformations that occur during the contrastive fine-tuning process have remained largely opaque. In "On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning," Xiao et al. (Durham University) investigate how CL re-architects the semantic space of pre-trained language models (PLMs), moving beyond simple performance metrics to analyze the structural evolution of embeddings.

The Challenge of Anisotropy

Standard pre-trained models, including BERT and RoBERTa, suffer from a phenomenon known as anisotropy. In these models, embeddings are not distributed uniformly across the vector space; instead, they occupy a "narrow cone." This representation degeneration results in high cosine similarity between even unrelated sentences, as the shared directional bias outweighs semantic distinctions. While contrastive learning—which optimizes for pulling similar pairs together and pushing dissimilar pairs apart—is known to mitigate this, the specific mechanics of this transformation require deeper inspection.

Methodology

The researchers analyzed state-of-the-art models, specifically all-mpnet-base-v2 and all-MiniLM-L6-v2, comparing them against their vanilla, non-contrastive counterparts. To quantify geometric shifts, they employed several diagnostic metrics:

  • Self-Similarity: Tracking how a token's embedding varies across different contexts.
  • Intra-Sentence Similarity: Measuring the convergence of tokens within the same sentence.
  • Dimension-level Inspection: Identifying "rogue dimensions" that disproportionately influence similarity calculations.
  • Ablation Training: Re-training models on the SNLI and MNLI datasets while varying hyperparameters such as temperature ($\tau$), batch size, and pooling strategies.

The "Entourage" Effect and Semantic Reorganization

One of the paper's most significant contributions is the identification of the "Entourage Effect." In vanilla PLMs, functional tokens (e.g., "the", "is", "of") maintain a high degree of independent contextualization. Contrastive learning fundamentally alters this relationship. Post-CL, functional tokens lose their independent semantic identity and begin to "follow" the semantically dense tokens (nouns and adjectives). They effectively become an entourage for the sentence's core meaning, moving in tandem with the primary semantic drivers within the vector space.

Furthermore, CL addresses "spurious contextualization." In vanilla models, meaningful words can become overly differentiated based on minor context shifts (e.g., "apple" the fruit vs. "Apple" the company). CL stabilizes the representations of these meaningful tokens while simultaneously increasing the contextualization of functional tokens, forcing the latter to align with the overall sentence intent.

Learning Dynamics: Temperature as "Thrust"

The study provides a mathematical and empirical look at the temperature hyperparameter ($\tau$). The authors characterize $\tau$ as a form of "thrust." A lower temperature increases the gradient magnitude for negative pairs, providing the necessary force to push embeddings apart and expand the representation space. This expansion is the primary driver for restoring isotropy. The authors suggest that when tuning models, searching for optimal temperature using a base of 2 (e.g., 0.025, 0.05, 0.1) is more effective than standard decimal increments, as it more precisely captures the exponential nature of the "thrust" required.

Key Results

The impact of CL on the geometry of the embedding space is profound:

  • Restoration of Isotropy: CL effectively eliminates the narrow cone problem. In vanilla MPNet, a single dimension accounted for 55% of the similarity metric. After contrastive training, it requires 209 dimensions to reach that same 50% threshold, indicating a significantly flatter and more expressive manifold.
  • Geometric Compression: Fine-tuning tends to push embeddings toward the origin, reducing the L2 norm. This "cleans" the space by minimizing the influence of poorly defined regions.
  • Intra-Sentence Convergence: CL drives tokens within a single sentence toward an extremely high similarity (~83%). This suggests that the model learns to apply intense cross-attention, squeezing all components of a sentence toward a singular, unique position in the semantic space.

Conclusion and Future Directions

Xiao et al. demonstrate that contrastive learning is not merely a refinement of existing representations but a fundamental restructuring of the semantic space. By enforcing isotropy and creating the "entourage effect," CL ensures that sentence embeddings are both distinct from one another and internally cohesive.

While this study focuses on the fine-tuning stage and its impact on Semantic Textual Similarity, the authors note that future research should investigate these dynamics during the pre-training phase. Additionally, the high intra-sentence similarity observed suggests potential implications for Information Retrieval (IR), where the ability to distinguish specific "signal" tokens from the "entourage" is vital.

Summary updated

BibTeX
@inproceedings{xiao2023isotropy,
  author = {Chenghao Xiao and Yang Long and Noura Al Moubayed},
  title = {{On Isotropy and Learning Dynamics of Contrastive-based Sentence Representation Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.778},
}
Figure