A Semantic Distance Metric Learning approach for Lexical Semantic Change Detection

Taichi Aida, Danushka Bollegala

Introduction

Language is inherently dynamic. A word like "cell" has transitioned from primarily denoting a prison room or biological unit to commonly referring to a mobile device. Detecting these shifts, known as Lexical Semantic Change Detection (SCD), is crucial for lexicography, sociology, and maintaining the relevance of Large Language Models (LLMs). Researchers have developed SDML (Semantic Distance Metric Learning), a supervised two-stage approach that significantly improves our ability to detect how word meanings shift over time.

The Challenges of Semantic Change Detection

Current SCD methods face two primary hurdles:

  1. The Representational Challenge: Words possess different meanings depending on context. Traditional methods often average word vectors across a corpus, which "conflates" distinct senses into a single, ambiguous representation.
  2. The Measurement Challenge: Standard distance functions like Cosine or Euclidean distance treat all vector dimensions as equally important and independent, failing to capture the complex semantic interactions between features.

Proposed Method: Semantic Distance Metric Learning (SDML)

The researchers propose a supervised, two-staged approach to address these challenges:

  • Stage 1: Learning a Sense-Aware Encoder. Instead of using raw pre-trained models, they fine-tune a Masked Language Model (MLM) using Word-in-Context (WiC) datasets. This teaches the model to specifically distinguish whether a target word carries the same or different meanings in two different sentences.
  • Stage 2: Learning a Sense-Aware Distance Metric. This is the core innovation. Rather than using standard Cosine distance, they learn a Mahalanobis distance metric. Using Information-Theoretic Metric Learning (ITML), they train a matrix that weights vector dimensions based on their relevance to semantic shifts and accounts for cross-dimensional correlations.
Comparison between conventional distance functions and the proposed sense-aware distance metric.

To compare a word between two time periods, the model calculates the average pairwise distance between every occurrence of the word in the first corpus and every occurrence in the second using this learned metric.

Experimental Setup and Datasets

The study utilizes a wide array of multilingual datasets for both training and evaluation:

  • Training (WiC Tasks): XL-WiC (German, French, Italian), MCL-WiC (English), and AM²iCo (covering 10 languages including Arabic, Japanese, and Finnish).
  • Evaluation (SCD Tasks): The gold-standard SemEval-2020 Task 1 (English, German, Swedish, Latin) and RuShiftEval (Russian).

Key Results and Insights

SDML achieved a 2-5% performance boost over previous state-of-the-art supervised and unsupervised baselines, with particularly strong results in English and German. Beyond historical change, the method set new benchmarks for the WiC task itself.

Performance comparison between the baseline sense-aware encoder and the full SDML approach across multiple datasets.

As shown in the results above, the addition of the sense-aware distance metric led to significant gains; for example, accuracy on the English MCL-WiC dataset jumped from 78.0% to 90.3%. Ablation studies confirmed that using the "full" Mahalanobis matrix (considering correlations between dimensions) significantly outperformed a simpler "diagonal" version.

Key Insights: * Supervision as a Proxy: While SCD research is often unsupervised, this paper proves that WiC data serves as an excellent proxy for training models to detect temporal change. * Cross-Lingual Transfer: The authors demonstrated that linguistic proximity allows for effective "weak supervision" across language families, such as using Italian data to detect changes in Latin.

Limitations and Future Work

Despite its success, SDML relies on the existence of WiC datasets, and performance can be lower for languages under-represented in pre-trained models (like Latin). Additionally, the pairwise comparison method is computationally intensive ($O(n_1 n_2)$). Future work will focus on expanding WiC datasets to more languages and exploring more efficient cross-lingual transfer methods.

Summary updated

Abstract

Detecting temporal semantic changes of words is an important task for various NLP applications that must make time-sensitive predictions. Lexical Semantic Change Detection (SCD) task involves predicting whether a given target word, $w$, changes its meaning between two different text corpora, $C_1$ and $C_2$. For this purpose, we propose a supervised two-staged SCD method that uses existing Word-in-Context (WiC) datasets. In the first stage, for a target word $w$, we learn two sense-aware encoders that represent the meaning of $w$ in a given sentence selected from a corpus. Next, in the second stage, we learn a sense-aware distance metric that compares the semantic representations of a target word across all of its occurrences in $C_1$ and $C_2$. Experimental results on multiple benchmark datasets for SCD show that our proposed method achieves strong performance in multiple languages. Additionally, our method achieves significant improvements on WiC benchmarks compared to a sense-aware encoder with conventional distance functions. Source code is available at https://github.com/LivNLP/svp-sdml .
BibTeX
@inproceedings{aida2024semantic,
  author = {Taichi Aida and Danushka Bollegala},
  title = {{A Semantic Distance Metric Learning approach for Lexical Semantic Change Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.451},
}
Figure