Detecting Lexical Semantic Change across Corpora with Smooth Manifolds (Student Abstract)

Anmol Goel, Ponnurangam Kumaraguru

Detecting Lexical Semantic Change across Corpora with Smooth Manifolds

Introduction

Language is a dynamic entity, with word meanings evolving over time or varying across different domains. This phenomenon, known as Lexical Semantic Change (LSC), is a central focus in historical linguistics and computational social science. For instance, the word "apple" has transitioned from denoting solely a fruit to frequently representing a global technology corporation. Detecting these shifts computationally involves comparing word embeddings—vector representations of words—trained on distinct corpora.

The Alignment Challenge

A fundamental hurdle in LSC detection is that word embeddings trained independently on different text collections exist in disparate coordinate systems. To compare them, they must first be aligned. The traditional approach, Orthogonal Procrustes (OP), attempts to rotate one vector space to match another within a flat Euclidean space. However, Euclidean geometry is often too restrictive to capture the complex, affine relationships between embedding spaces. Furthermore, maintaining orthogonality during optimization in Euclidean space is computationally cumbersome due to strict constraints.

A Manifold-Based Approach

To overcome these limitations, researchers have proposed a novel geometric framework that treats alignment as an optimization task on smooth Riemannian manifolds. Instead of a rigid Euclidean alignment, this method utilizes two specific manifold types:

  1. Stiefel Manifolds: These are used to learn corpus-specific orthogonal transformations ($P$ and $Q$). By optimizing on the Stiefel manifold, the model ensures that transformations remain orthogonal by design, effectively turning a constrained problem into an unconstrained one.
  2. Positive Definite Manifolds: These are employed to learn a Mahalanobis metric ($M$). Unlike standard cosine similarity, which treats all vector dimensions equally, the Mahalanobis metric accounts for feature correlations, allowing the model to weight contextually significant aspects of a word's meaning more heavily.
The proposed framework maps embeddings from separate spaces A and B into a shared latent space using transformations learned on smooth manifolds.

By rotating both embedding spaces into a shared latent space and simultaneously learning a feature-scaling metric, the model captures nuanced semantic relationships that traditional methods miss.

Experimental Evaluation

The proposed model was evaluated using the Diachronic Usage Relatedness (DURel) dataset, which consists of 22 German words from the Deutsches Textarchiv (DTA). Human annotators provided ground-truth semantic change scores on a scale of 1 to 4. Word2vec embeddings (300 dimensions) were trained on two historical splits of the DTA: 1750–1799 and 1850–1899.

Results

The performance was measured using Spearman’s rank correlation, comparing predicted change scores against human judgments. The results demonstrate a significant performance leap over previous state-of-the-art methods.

Comparison of Spearman correlation scores on the DURel dataset. The manifold-based approach significantly outperforms the baseline established by Gonen et al. (2020).

The manifold approach achieved a Spearman score of 0.77, compared to 0.59 for the "Stable Word Neighbors" method (Gonen et al., 2020). This substantial improvement suggests that the structural bias inherent in smooth manifolds is exceptionally well-suited for modeling the latent relationships between language models.

Conclusion and Future Work

By leveraging the geometry of smooth manifolds, this framework provides a more sensitive "thermometer" for detecting semantic shifts. Future research directions include moving toward fully unsupervised regimes to eliminate the need for stability assumptions and extending the framework to contextual embeddings like BERT, which could offer even more granular insights into how word usage evolves within specific contexts.

Summary updated

Abstract

Comparing two bodies of text and detecting words with significant lexical semantic shift between them is an important part of digital humanities. Traditional approaches have relied on aligning the different embeddings using the Orthogonal Procrustes problem in the Euclidean space. This study presents a geometric framework that leverages smooth Riemannian manifolds for corpus-specific orthogonal rotations and a corpus-independent scaling metric to project the different vector spaces into a shared latent space. This enables us to capture any affine relationship between the embedding spaces while utilising the rich geometry of smooth manifolds.
BibTeX
@inproceedings{goel2021detecting,
  author = {Anmol Goel and Ponnurangam Kumaraguru},
  title = {{Detecting Lexical Semantic Change across Corpora with Smooth Manifolds (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17888},
}
Figure