TL;DR
This paper introduces a novel supervised approach, Semantic Distance Metric Learning (SDML), for detecting how word meanings change over time. By learning both sense-aware word representations and a specialized distance metric, SDML significantly outperforms existing methods on several benchmark datasets, demonstrating the importance of capturing nuanced word meanings.
What Problem Does This Paper Address?
Understanding how word meanings evolve is crucial for many NLP applications, from lexicography to improving large language models (LLMs). The "Lexical Semantic Change Detection" (SCD) task aims to identify if a word's meaning has shifted between two different time periods (represented by text corpora $C_1$ and $C_2$). The core challenges are:
- Representational Challenge: Words can have multiple meanings (polysemy) even within the same corpus. Simply averaging word embeddings across a corpus can conflate these distinct senses, making it hard to track genuine semantic shifts.
- Measurement Challenge: Accurately measuring the semantic difference between word representations from different time periods is difficult, especially when labeled data for SCD is scarce, forcing reliance on unsupervised methods and standard distance metrics (like cosine or Euclidean) that might not capture subtle semantic differences.
Methods
The authors propose a two-stage supervised approach called Semantic Distance Metric Learning (SDML):
- Stage 1: Sense-Aware Encoder Learning: They leverage existing Word-in-Context (WiC) datasets, which are annotated for whether a word has the same or different meaning in two sentences. Using a Siamese bi-encoder architecture (based on a pretrained MLM like XLM-ROBERTa), they fine-tune the encoder to produce representations that are sensitive to word senses. This addresses the representational challenge by learning to distinguish between different meanings of a word.
- Stage 2: Sense-Aware Distance Metric Learning: After obtaining sense-aware embeddings for a target word in sentences from $C_1$ and $C_2$, they learn a specialized sense-aware distance metric. This metric, a Mahalanobis distance, is trained using the WiC data to assign smaller distances to sentences where the target word has the same meaning and larger distances when meanings differ. Unlike standard metrics, Mahalanobis distance accounts for inter-dimensional correlations and varying importance of different dimensions, offering a more nuanced comparison.
The final semantic change score for a word between $C_1$ and $C_2$ is computed by averaging the sense-aware distances between all pairs of sentence representations for that word from the two corpora.
Datasets
The paper utilizes several datasets:
- Training:
- XL-WiC: A multilingual WiC dataset.
- MCL-WiC: Another multilingual WiC dataset.
-
AM2iCo: A WiC dataset for low-resource languages. These WiC datasets provide the necessary sense-level supervision for training both the encoder and the distance metric.
-
Evaluation (SCD Benchmarks):
- SemEval-2020 Task 1: A benchmark for lexical semantic change detection across English, German, Swedish, and Latin.
- RuShiftEval: A benchmark for Russian semantic change detection.
Key Insights and Ideas
- WiC as a Proxy for SCD Supervision: The paper cleverly uses WiC datasets, designed for word sense disambiguation, as a source of supervision for SCD. This is crucial because directly labeled SCD datasets are scarce.
- Mahalanobis Distance for Nuance: The adoption of Mahalanobis distance is a key innovation. It moves beyond simple cosine or Euclidean distances by learning a metric that weights dimensions and captures correlations, which is vital for accurately comparing potentially subtle semantic shifts.
- Two-Stage Learning is Crucial: The success of SDML hinges on both learning good sense-aware representations and a good distance metric. The ablation studies show that either component alone is less effective than their combination.
- Low-Resource Language Performance: SDML shows promising results even for languages with limited WiC training data, suggesting its robustness and potential for broader applicability.
Key Results
- SCD Performance: SDML achieves state-of-the-art or highly competitive results on SemEval-2020 Task 1 and RuShiftEval benchmarks, showing significant improvements (2-5%) over strong baselines.
- WiC Performance: SDML also demonstrates significant improvements on WiC benchmarks across multiple languages, validating its ability to discriminate between same and different word senses.
- Ablation Study: Using the full Mahalanobis matrix (SDMLfull) provides further improvements over using only diagonal components (SDMLdiag), highlighting the importance of capturing cross-dimensional correlations.
- Polysemy and Frequency: The study found that polysemous words are more likely to change meaning, while word frequency does not show a consistent correlation with semantic change, aligning with prior observations.
Limitations and Future Work
- Language Coverage: While SDML performs well in English, German, and Russian, its performance in low-resource languages like Swedish and Latin is limited due to the lack of corresponding WiC training data. The authors suggest exploring cross-lingual transfer methods to address this.
- Latin Performance: The performance on Latin was notably lower, attributed to its limited presence in both the pre-training data of the underlying MLM and the WiC datasets used for fine-tuning.
- Ethical Considerations: The authors acknowledge that publicly available MLMs can encode social biases, and these biases might be amplified during the sense-aware encoder training. They emphasize the need to evaluate for such biases before downstream use.
- Future Work: Expanding WiC datasets to include more languages is suggested to improve performance for under-represented languages. Further investigation into the impact of frequency and polysemy on semantic change is also proposed.
Summary updated