BOS at LSCDiscovery: Lexical Substitution for Interpretable Lexical Semantic Change Detection

Artem Kudisov, Nikolay ArefyevACL 2022

TL;DR

This paper introduces a novel approach for detecting and interpreting lexical semantic change in Spanish by generating "bag-of-substitutes" (BOS) representations. Their method achieves the second-best results in sense loss and sense gain detection, offering a more interpretable way to understand how word meanings evolve over time.

What Problem Does This Paper Address?

Understanding how the meaning of words changes over time (lexical semantic change, or LSC) is crucial for historical linguistics, digital humanities, and even tracking the evolution of language in online communities. Traditional methods often struggle with interpretability, making it difficult to pinpoint why a word's meaning has shifted. The LSCDiscovery shared task specifically challenged researchers to not only detect semantic change but also to identify which senses were gained or lost. This paper tackles the challenge of providing a more transparent and interpretable solution to LSC detection, moving beyond simple "change/no change" predictions.

Methods

The core of the proposed solution lies in the Bag-of-Substitutes (BOS) representation. Instead of relying on the actual words present in a text fragment (like traditional bag-of-words), BOS uses words that can grammatically and semantically substitute the target word within its context. This approach is powered by the XLM-R masked language model, pre-trained on a massive multilingual dataset.

The process involves:

  1. Collecting Data: Gathering examples of a target word from two distinct time periods (e.g., 1810-1906 and 1994-2020).
  2. Substitute Generation: Using XLM-R with specific "dynamic patterns" (similar to Hearst patterns) to generate a set of substitutes for each example. These patterns help guide the model to produce semantically related words (like synonyms or hypernyms) rather than just any plausible word.
  3. BOS Vector Creation: For each time period, a BOS vector is created by aggregating the probabilities of the generated substitutes across all examples.
  4. Change Detection: The Average Pairwise Distance (APD) between the BOS vectors of the old and new time periods is calculated. A larger APD suggests a greater semantic shift.
  5. Interpretability: The key innovation is that by examining the specific substitutes that are prevalent in one time period but absent in another, researchers can gain insights into which senses were acquired or lost.

Datasets

The paper uses Spanish corpora from two distinct time periods:

  • 1810-1906: The older corpus.
  • 1994-2020: The modern corpus.

These corpora were provided as part of the LSCDiscovery shared task. The authors also mention using a development set of 20 words for tuning weights.

Key Insights and Ideas

  • Substitutes as Meaning Proxies: The fundamental idea that words which can substitute a target word in context are strong indicators of its meaning is powerful. This moves beyond simple co-occurrence statistics.
  • Interpretable Change Detection: The ability to identify specific substitutes that appear or disappear across time periods is a significant step towards understanding why a word's meaning has changed, rather than just knowing that it has.
  • Dynamic Patterns Matter: The use of "dynamic patterns" with the masked language model is crucial for generating semantically relevant substitutes, going beyond random predictions.
  • Two-Subword Substitutes Outperform One-Subword: The experiments suggest that using patterns that generate two-subword substitutes generally leads to better results than single-subword substitutes.
  • Brackets and Dashes Can Hinder: Surprisingly, removing brackets and dashes from the substitution patterns improved results, suggesting these punctuation marks might interfere with the model's ability to generate accurate substitutes in Spanish.

Key Results

  • The proposed BOS-based approach achieved the second-best result in the sense loss and sense gain detection subtasks of the LSCDiscovery shared task.
  • For the graded change discovery task (ranking words by degree of change), their post-evaluation results were comparable to the 3rd best participant.
  • The model using two-subword substitutes (LS_m2_2) showed better performance than one-subword substitutes (LS_m1_2).
  • Combining multiple patterns (LS_m1_7) led to significant improvements, even with somewhat arbitrarily selected weights.

Limitations and Future Work

  • Suboptimal Weights: The weights for combining different dynamic patterns were selected based on a small development set and are acknowledged as likely suboptimal. Future work could focus on developing more principled methods for finding optimal patterns and weights.
  • Computational Cost: Generating substitutes for a large number of examples is computationally intensive, requiring significant GPU resources.
  • Ambiguity in Spanish: The paper notes that the effectiveness of certain patterns might be influenced by the specificities of the Spanish language, such as the role of brackets and dashes.
  • Further Exploration of Patterns: The authors suggest that exploring more dynamic patterns and their combinations could lead to further improvements.

Summary updated

BibTeX
@inproceedings{kudisov2022bos,
  author = {Artem Kudisov and Nikolay Arefyev},
  title = {{BOS at LSCDiscovery: Lexical Substitution for Interpretable Lexical Semantic Change Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure