Semantic Geometry of Sentence Embeddings

Matthieu TehenanEMNLP 2025

TL;DR

This paper introduces a novel framework to understand sentence embeddings not just as points in space, but as organized geometric structures governed by information theory. They demonstrate that these embeddings exhibit a "semantic geometry" where features correspond to directions and concepts form regions, enabling more interpretable and controllable NLP models.

What Problem Does This Paper Address?

Modern NLP heavily relies on sentence embeddings – vector representations that capture sentence meaning. While these embeddings power tasks like semantic search and question answering, they remain largely "black boxes." It's difficult to understand why two sentences are considered similar or dissimilar, and users have little control over how these embeddings behave for specific downstream tasks. This lack of interpretability hinders not only performance improvements but also crucial aspects like transparency and safety in AI systems. Existing methods often reduce sentence similarity to a single score, obscuring the rich internal structure that could be leveraged.

Methods

The core of this work is a formal framework that connects sentence embeddings to information theory, specifically pointwise mutual information (PMI). They propose that the inner product between sentence embeddings (after normalization) approximates PMI. This allows them to reframe the embedding space as a "semantic geometry."

Key methodological contributions include:

  • Formal Framework: Establishing the link between contrastive learning objectives (common in training sentence embeddings) and PMI.
  • Feature Identification: Proposing that features correspond to directions on the hypersphere. They leverage the assumption that sentences sharing a feature form a region that can be modeled by a von Mises-Fisher distribution, allowing them to estimate feature directions as normalized sample means.
  • Compositionality: Showing that linear combinations of these feature directions correspond to composed semantic features, mirroring how linguistic meaning is built from parts. This allows for operations like combining "South America" and "North America" into "America."
  • Semantic Regions: Defining semantic regions on the hypersphere corresponding to the presence of specific features. These regions can exhibit set-theoretic relationships (inclusion, exclusion, overlap) that reflect semantic relationships (e.g., "corgi" is a subset of "dog").

Datasets

The paper uses two datasets for evaluation:

  • WordNet: A synthetically generated dataset based on the WordNet lexical database. It's constructed by creating sentences that incorporate hypernyms and hyponyms, aiming to test how semantic hierarchies are represented.
  • TREC: A real-world dataset for question classification, used to evaluate how hierarchical semantic structures are preserved in embeddings.

Key Insights and Ideas

  • Embeddings as Information Proxies: Sentence embeddings, particularly when trained with contrastive objectives, are not just arbitrary vectors but encode information-theoretic relationships (PMI) between sentences.
  • Geometric Structure of Meaning: The embedding space has an inherent "semantic geometry." Features correspond to specific directions, and concepts or categories form distinct regions on the hypersphere.
  • Compositionality is Geometric: The way linguistic meaning is composed from smaller parts is reflected in the geometric operations (linear combinations) on feature directions in the embedding space.
  • Hierarchical Relationships are Encoded: Semantic hierarchies (like hypernym-hyponym relationships) are reflected in the geometric arrangement of embeddings, with hypernyms often being closer to centroids or exhibiting specific ordering patterns.
  • Interpretability through Geometry: By understanding this geometry, we can move beyond black-box models to identify interpretable features and control embedding behavior.

Key Results

  • Hierarchical Preservation: Across several models and datasets (WordNet, TREC), the embeddings generally preserve hierarchical semantic relationships, with hypernyms showing expected relationships to hyponyms and centroids. For instance, in the WordNet dataset, hypernym sentences are consistently closer to the centroid of their semantic category.
  • Semantic Regions Exist: Experiments on the WordNet dataset show that prototype sentences for a category (e.g., "dog") are more similar to their hyponyms than hyponyms are to each other, supporting the idea of semantic regions.
  • Efficiency of Centroid Method: The proposed centroid-based interpretability method is significantly more efficient (1-2 orders of magnitude) than linear probing, requiring no parameter training and only a single forward pass. It achieves competitive accuracy, especially for fine-grained classification.
  • Model Disparities: Smaller models (like all-MiniLM-L6-v2) show a more compressed geometric representation, suggesting limitations in their ability to encode complex semantic structures compared to larger models.

Limitations and Future Work

  • Dataset Size and Complexity: The current dataset size poses a challenge for comprehensively identifying all semantic sets. Future work should aim to map principal sets that drive overall structure and interpretability.
  • Model Nuance: A simple model might not fully capture the nuances of embedding variability and uncertainty. Future research could explore models that better represent these aspects.
  • Mapping to Principal Sets: The paper suggests that mapping embeddings to principal semantic sets is a key direction for future research.
  • Broader Applications: The uncovered geometry has potential relevance beyond NLP, echoing cognitive theories of similarity and conceptual spaces.

Summary updated

BibTeX
@inproceedings{tehenan2025semantic,
  author = {Matthieu Tehenan},
  title = {{Semantic Geometry of Sentence Embeddings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure