XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE

Pierluigi Cassotti, Lucia Siciliani, Marco de Gemmis, Giovanni Semeraro, Pierpaolo BasileACL 2023 · DOI

Abstract

Lexical Semantic Change (LSC) detection is the task of identifying shifts in word meanings over time. While cross-encoders currently define the state-of-the-art, their computational cost limits scalability. We present XL-LEXEME, a bi-encoder model pre-trained on large-scale Word-in-Context (WiC) datasets. XL-LEXEME achieves state-of-the-art performance on English, German, Swedish, and Russian LSC benchmarks, outperforming more complex cross-encoder architectures while remaining significantly more efficient.

Introduction

The meanings of words are fluid, often shifting dramatically across decades or centuries (e.g., "mouse," "broadcast," "apple"). Automatically detecting these shifts, known as Lexical Semantic Change (LSC) detection, is crucial for historical linguistics and social science.

The current gold standard for comparing word meanings in different contexts is the cross-encoder, which processes sentence pairs simultaneously. However, cross-encoders are computationally expensive and do not generate independent, comparable embeddings for individual words. This paper introduces XL-LEXEME, a bi-encoder model that leverages "Word-in-Context" (WiC) tasks to learn robust semantic representations that transfer effectively to diachronic (historical) settings.

Methodology

XL-LEXEME is built on the Sentence-BERT (SBERT) framework, utilizing a Siamese Network architecture based on XLM-RoBERTa Large.

Architecture and Representation

Unlike cross-encoders, XL-LEXEME encodes two sentences separately. To focus the model on specific target words, the authors employ Target Highlighting, wrapping the target word in special delimiters: <t> target </t>. The final representation of a word in context is derived from the sum of the encoded subword vectors located between these delimiters.

Training and Transfer Learning

The model is trained using Contrastive Loss, learning to minimize the distance between embeddings when a word shares the same meaning across contexts and maximize it when meanings differ. A key innovation is the use of synchronic transfer learning: the model is trained on modern WiC datasets (where meanings are known) and then applied to diachronic data (sentences from different time periods) to detect historical shifts.

Datasets

The model was pre-trained on a merger of massive WiC datasets, including MCL-WiC, AM$^2$ICO, and XL-WiC. Evaluation was conducted on two major LSC benchmarks:

  • SemEval-2020 Task 1: Covering English (EN), German (DE), Swedish (SV), and Latin (LA).
  • RuShiftEval: A Russian shared task involving three distinct time periods (pre-Soviet, Soviet, and post-Soviet).

Results and Discussion

XL-LEXEME established new benchmarks across multiple languages. On the SemEval-2020 leaderboard, it achieved the highest Spearman correlation scores for English (0.757), German (0.877), and Swedish (0.754).

SemEval-2020 Task 1 Subtask 2 Leaderboard results.

In the RuShiftEval task, XL-LEXEME (fine-tuned) obtained state-of-the-art results for Russian, with an average correlation of 0.825.

RuShiftEval Leaderboard results for Russian LSC detection.

Efficiency and Cross-Lingual Transfer

A significant finding is that XL-LEXEME outperforms cross-encoders while maintaining the efficiency of a bi-encoder ($O(N^2)$ vs $O((N/2)^2)$ complexity). Furthermore, the model demonstrated strong cross-lingual transfer; for instance, it performed exceptionally well on Swedish despite Swedish not being present in the WiC training set, likely benefiting from XLM-RoBERTa's underlying cross-lingual knowledge.

Limitations

The model struggled with Latin (-0.056 correlation), likely due to Latin's underrepresentation in the base XLM-R model and the significant linguistic distance between ancient Latin and modern training data. Additionally, the small size of evaluation sets (31–48 words) suggests a need for larger-scale benchmarks in future work.

Conclusion

XL-LEXEME demonstrates that a bi-encoder model trained on synchronic word-in-context data can effectively and efficiently detect diachronic lexical semantic change. By setting new state-of-the-art results across several languages, it provides a scalable path forward for large-scale historical semantic analysis.

Reference: Cassotti, P., et al. (2023). XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE. Proceedings of ACL 2023.

Summary updated

BibTeX
@inproceedings{cassotti2023xl,
  author = {Pierluigi Cassotti and Lucia Siciliani and Marco de Gemmis and Giovanni Semeraro and Pierpaolo Basile},
  title = {{XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-short.135},
}
Figure