Introduction
Language is inherently dynamic. A word like "cell" has transitioned from primarily denoting a prison room or biological unit to commonly referring to a mobile device. Detecting these shifts, known as Lexical Semantic Change Detection (SCD), is crucial for lexicography, sociology, and maintaining the relevance of Large Language Models (LLMs). Researchers have developed SDML (Semantic Distance Metric Learning), a supervised two-stage approach that significantly improves our ability to detect how word meanings shift over time.
The Challenges of Semantic Change Detection
Current SCD methods face two primary hurdles:
- The Representational Challenge: Words possess different meanings depending on context. Traditional methods often average word vectors across a corpus, which "conflates" distinct senses into a single, ambiguous representation.
- The Measurement Challenge: Standard distance functions like Cosine or Euclidean distance treat all vector dimensions as equally important and independent, failing to capture the complex semantic interactions between features.
Proposed Method: Semantic Distance Metric Learning (SDML)
The researchers propose a supervised, two-staged approach to address these challenges:
- Stage 1: Learning a Sense-Aware Encoder. Instead of using raw pre-trained models, they fine-tune a Masked Language Model (MLM) using Word-in-Context (WiC) datasets. This teaches the model to specifically distinguish whether a target word carries the same or different meanings in two different sentences.
- Stage 2: Learning a Sense-Aware Distance Metric. This is the core innovation. Rather than using standard Cosine distance, they learn a Mahalanobis distance metric. Using Information-Theoretic Metric Learning (ITML), they train a matrix that weights vector dimensions based on their relevance to semantic shifts and accounts for cross-dimensional correlations.
To compare a word between two time periods, the model calculates the average pairwise distance between every occurrence of the word in the first corpus and every occurrence in the second using this learned metric.
Experimental Setup and Datasets
The study utilizes a wide array of multilingual datasets for both training and evaluation:
- Training (WiC Tasks): XL-WiC (German, French, Italian), MCL-WiC (English), and AM²iCo (covering 10 languages including Arabic, Japanese, and Finnish).
- Evaluation (SCD Tasks): The gold-standard SemEval-2020 Task 1 (English, German, Swedish, Latin) and RuShiftEval (Russian).
Key Results and Insights
SDML achieved a 2-5% performance boost over previous state-of-the-art supervised and unsupervised baselines, with particularly strong results in English and German. Beyond historical change, the method set new benchmarks for the WiC task itself.
As shown in the results above, the addition of the sense-aware distance metric led to significant gains; for example, accuracy on the English MCL-WiC dataset jumped from 78.0% to 90.3%. Ablation studies confirmed that using the "full" Mahalanobis matrix (considering correlations between dimensions) significantly outperformed a simpler "diagonal" version.
Key Insights: * Supervision as a Proxy: While SCD research is often unsupervised, this paper proves that WiC data serves as an excellent proxy for training models to detect temporal change. * Cross-Lingual Transfer: The authors demonstrated that linguistic proximity allows for effective "weak supervision" across language families, such as using Italian data to detect changes in Latin.
Limitations and Future Work
Despite its success, SDML relies on the existence of WiC datasets, and performance can be lower for languages under-represented in pre-trained models (like Latin). Additionally, the pairwise comparison method is computationally intensive ($O(n_1 n_2)$). Future work will focus on expanding WiC datasets to more languages and exploring more efficient cross-lingual transfer methods.
Summary updated