GlossReader at LSCDiscovery: Train to Select a Proper Gloss in English – Discover Lexical Semantic Change in Spanish

Maxim Rachinskiy, Nikolay ArefyevACL 2022

TL;DR

This paper tackles the challenge of detecting lexical semantic change (LSC) in Spanish by fine-tuning a multilingual masked language model (XLM-R) on a Word Sense Disambiguation (WSD) task. This approach effectively mitigates the "word form bias" in standard LMs, leading to state-of-the-art results on the LSCDiscovery shared task for Spanish.

What Problem Does This Paper Address?

Lexical Semantic Change Detection (LSCD) aims to automatically identify how the meaning of words evolves over time. Traditional approaches using contextualized embeddings from language models (LMs) struggle with a "word form bias," meaning they are more sensitive to changes in word spelling or grammatical form than to actual shifts in meaning. This makes them less effective for LSCD. Furthermore, creating large, annotated datasets for LSCD is extremely labor-intensive, with existing datasets often containing fewer than 100 labeled words, which is insufficient for fine-tuning complex models. The LSCDiscovery shared task specifically focuses on detecting semantic change in Spanish texts from two distinct time periods (1810-1906 and 1994-2020).

Methods

The core innovation of this work lies in a novel fine-tuning strategy. Instead of directly fine-tuning a masked language model (MLM) like XLM-R on LSCD data (which is scarce), the authors fine-tune it on a large English Word Sense Disambiguation (WSD) dataset. They adapt the Bi-Encoder Model (BEM) architecture, which uses separate encoders for word contexts and word senses (glosses). By replacing the English BERT in the original BEM with the multilingual XLM-R, they create a multilingual WSD system. This system is trained on English WSD data, and then its cross-lingual transferability is leveraged to generate contextualized embeddings for Spanish words without requiring any Spanish WSD data.

For LSCD, they calculate the average distance between these improved contextualized embeddings of a word's occurrences in the old and new corpora. This "Average Pairwise Distance" (APD) serves as a graded measure of semantic change. For binary change detection, they apply thresholding to these APD scores.

Datasets

  • Training: The WSD system was trained on the English WSD dataset SemCor, which contains over 200K sense-annotated word occurrences. The SemEval-2007 WSD dataset was used as a development set.
  • Evaluation: The LSCDiscovery shared task in Spanish was used for evaluation, comprising two corpora representing different time periods. The task included graded change detection and binary change detection subtasks.

Key Insights and Ideas

  • Mitigating Word Form Bias: The key insight is that fine-tuning an MLM on a task like WSD, which inherently requires understanding word senses and can downplay word forms, effectively reduces the problematic word form bias.
  • Cross-Lingual Transfer for LSCD: Leveraging the zero-shot cross-lingual capabilities of XLM-R allows for effective LSCD in a low-resource language (Spanish) by training on a high-resource language (English) for a related task (WSD).
  • WSD as a Pretext Task for LSCD: Using WSD as a pretext task for fine-tuning LMs proves to be a highly effective strategy for improving their performance on LSCD, even without direct LSCD fine-tuning.
  • APD as a Proxy for Semantic Similarity: The Average Pairwise Distance (APD) derived from normalized contextualized embeddings serves as a reasonable approximation of semantic similarity between word occurrences across corpora, aligning well with metrics like COMPARE.

Key Results

The proposed GlossReader system achieved state-of-the-art results on the LSCDiscovery shared task, outperforming all other participants in both graded change detection subtasks. It also achieved top results in two out of the three binary change detection subtasks. The paper highlights that fine-tuning XLM-R on the WSD task provided a significant performance boost compared to using the original, pre-trained XLM-R.

Limitations and Future Work

  • Sense Gain Detection: The authors acknowledge that their solution did not specifically address the optional sense gain detection subtask and performed less competitively there. They suggest that this subtask might require dedicated approaches.
  • APD vs. JSD: While APD correlates well with the gold COMPARE scores, it is noted that it might be sub-optimal for approximating the gold Jensen-Shannon Distance (JSD) scores. Future work could focus on developing methods to specifically approximate JSD.
  • Thresholding Strategy: For binary change detection, the authors experimented with two thresholding strategies and submitted the same predictions for all binary subtasks. They suggest that optimizing these strategies could lead to further improvements.

Summary updated

BibTeX
@inproceedings{rachinskiy2022glossreader,
  author = {Maxim Rachinskiy and Nikolay Arefyev},
  title = {{GlossReader at LSCDiscovery: Train to Select a Proper Gloss in English – Discover Lexical Semantic Change in Spanish}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure