Abstract
Lexical Semantic Change (LSC) detection is the task of identifying shifts in word meanings over time. While cross-encoders currently define the state-of-the-art, their computational cost limits scalability. We present XL-LEXEME, a bi-encoder model pre-trained on large-scale Word-in-Context (WiC) datasets. XL-LEXEME achieves state-of-the-art performance on English, German, Swedish, and Russian LSC benchmarks, outperforming more complex cross-encoder architectures while remaining significantly more efficient.
Introduction
The meanings of words are fluid, often shifting dramatically across decades or centuries (e.g., "mouse," "broadcast," "apple"). Automatically detecting these shifts, known as Lexical Semantic Change (LSC) detection, is crucial for historical linguistics and social science.
The current gold standard for comparing word meanings in different contexts is the cross-encoder, which processes sentence pairs simultaneously. However, cross-encoders are computationally expensive and do not generate independent, comparable embeddings for individual words. This paper introduces XL-LEXEME, a bi-encoder model that leverages "Word-in-Context" (WiC) tasks to learn robust semantic representations that transfer effectively to diachronic (historical) settings.
Methodology
XL-LEXEME is built on the Sentence-BERT (SBERT) framework, utilizing a Siamese Network architecture based on XLM-RoBERTa Large.
Architecture and Representation
Unlike cross-encoders, XL-LEXEME encodes two sentences separately. To focus the model on specific target words, the authors employ Target Highlighting, wrapping the target word in special delimiters: <t> target </t>. The final representation of a word in context is derived from the sum of the encoded subword vectors located between these delimiters.
Training and Transfer Learning
The model is trained using Contrastive Loss, learning to minimize the distance between embeddings when a word shares the same meaning across contexts and maximize it when meanings differ. A key innovation is the use of synchronic transfer learning: the model is trained on modern WiC datasets (where meanings are known) and then applied to diachronic data (sentences from different time periods) to detect historical shifts.
Datasets
The model was pre-trained on a merger of massive WiC datasets, including MCL-WiC, AM$^2$ICO, and XL-WiC. Evaluation was conducted on two major LSC benchmarks:
- SemEval-2020 Task 1: Covering English (EN), German (DE), Swedish (SV), and Latin (LA).
- RuShiftEval: A Russian shared task involving three distinct time periods (pre-Soviet, Soviet, and post-Soviet).
Results and Discussion
XL-LEXEME established new benchmarks across multiple languages. On the SemEval-2020 leaderboard, it achieved the highest Spearman correlation scores for English (0.757), German (0.877), and Swedish (0.754).
In the RuShiftEval task, XL-LEXEME (fine-tuned) obtained state-of-the-art results for Russian, with an average correlation of 0.825.
Efficiency and Cross-Lingual Transfer
A significant finding is that XL-LEXEME outperforms cross-encoders while maintaining the efficiency of a bi-encoder ($O(N^2)$ vs $O((N/2)^2)$ complexity). Furthermore, the model demonstrated strong cross-lingual transfer; for instance, it performed exceptionally well on Swedish despite Swedish not being present in the WiC training set, likely benefiting from XLM-RoBERTa's underlying cross-lingual knowledge.
Limitations
The model struggled with Latin (-0.056 correlation), likely due to Latin's underrepresentation in the base XLM-R model and the significant linguistic distance between ancient Latin and modern training data. Additionally, the small size of evaluation sets (31–48 words) suggests a need for larger-scale benchmarks in future work.
Conclusion
XL-LEXEME demonstrates that a bi-encoder model trained on synchronic word-in-context data can effectively and efficiently detect diachronic lexical semantic change. By setting new state-of-the-art results across several languages, it provides a scalable path forward for large-scale historical semantic analysis.
Reference: Cassotti, P., et al. (2023). XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE. Proceedings of ACL 2023.
Summary updated