LSCDiscovery: A shared task on semantic change discovery and detection in Spanish

Frank D. Zamora-Reina, Felipe Bravo-Marquez, Dominik SchlechtwegACL 2022

TL;DR

This paper introduces the first shared task focused on discovering and detecting semantic change in Spanish, a language spoken by over 450 million people. By creating a novel, large-scale dataset and organizing a competition, the authors push the boundaries of computational linguistics to understand how word meanings evolve over time, a crucial task for historical linguistics and lexicography.

What Problem Does This Paper Address?

Understanding how the meaning of words changes over time (Lexical Semantic Change Detection or LSCD) is a fundamental challenge in linguistics and computational linguistics. While previous research has organized shared tasks for languages like English and German, Spanish has been largely overlooked, despite its global significance. Furthermore, prior tasks often focused on small, pre-selected sets of words, which doesn't reflect the real-world need to analyze entire vocabularies. This paper addresses these gaps by creating a comprehensive dataset for Spanish and designing a task that encourages discovery across a broader range of words.

Methods

The core of this work is a shared task with two main phases:

  1. Graded Change Discovery: Participants had to rank all content words in a Spanish corpus based on how much their meaning changed between two time periods (1810-1906 and 1994-2020). This is a challenging "discovery" task as it requires predictions for a vast number of words without prior selection.
  2. Binary Change Detection: This phase focused on a smaller, pre-selected set of words, asking participants to classify whether a word had undergone a significant semantic change or not.

To evaluate these tasks, the authors created a new dataset by manually annotating word usages. They used a framework where annotators judged the semantic relatedness of word usages, building "Word Usage Graphs" (WUGs). These graphs were then clustered to infer word senses, and changes in these clusters over time were used to derive gold-standard labels for semantic change.

Datasets

The paper introduces a new benchmark dataset for Spanish semantic change. This dataset is based on two large Spanish corpora:

  • Old Corpus (C1): 1810-1906 (approx. 13 million tokens)
  • Modern Corpus (C2): 1994-2020 (approx. 22 million tokens)

For the shared task, they created:

  • Public Target Words: A list of 4385 words from the intersection of the corpora's vocabularies.
  • Hidden Target Words: A subset of these words (100 words) were manually annotated for semantic change. This annotated data was used for evaluation, while participants worked with the full corpora for the "discovery" aspect.

The annotation process involved 12 native Spanish speakers and resulted in approximately 62,000 semantic proximity judgments.

Key Insights and Ideas

  • The "Discovery" Challenge is Crucial: The paper highlights that true semantic change discovery, requiring analysis of entire vocabularies, is a more realistic and important task for lexicography than just detecting change in pre-selected words.
  • Token-Based Models Dominate: For Spanish, token-based models (which analyze individual word instances in context) outperformed type-based models (which analyze words as abstract units) in the shared task, continuing a trend seen in previous research.
  • Clustering Approaches Gain Traction: For the first time in LSCD shared tasks, clustering-based methods emerged as winning approaches, suggesting a promising direction for future research.
  • Modeling the Annotation Procedure Matters: Top-performing systems often didn't directly model the human annotation process (e.g., using JSD on cluster distributions). Instead, they leveraged correlations between different change scores (like COMPARE and JSD), indicating a potential upper bound on their performance.
  • Spanish is a Rich Area for LSCD: The success of the shared task demonstrates the viability and importance of studying semantic change in Spanish.

Key Results

  • Graded Change Discovery: The winning system achieved a Spearman rank correlation of 0.735, comparable to top performance in previous English tasks.
  • Binary Change Detection: The best system achieved an F1 score of 0.716. While lower than Graded Change, this was still significantly better than baseline methods.
  • System Performance: GlossReader and DeepMistake were top performers in Graded Change Discovery and COMPARE, while GlossReader also led in Binary Change Detection.
  • Baselines: The SGNS+OP+CD baseline performed well, but was outperformed by the top human-driven systems.

Limitations and Future Work

  • Binary Change Detection Performance: The paper notes that Binary Change Detection still lags behind Graded Change, suggesting room for improvement.
  • Upper Bound of Current Methods: The success of systems that model correlations rather than the annotation procedure itself indicates a need for new approaches to break performance ceilings.
  • Future Directions: The authors hope this shared task and dataset will spur further research into LSCD for Spanish, encouraging the development of novel techniques, particularly those that better model the human annotation process, potentially through advanced clustering methods. They also suggest that the data can be used for proposing new ideas and techniques in the field.

Summary updated

BibTeX
@inproceedings{zamorareina2022lscdiscovery,
  author = {Frank D. Zamora-Reina and Felipe Bravo-Marquez and Dominik Schlechtweg},
  title = {{LSCDiscovery: A shared task on semantic change discovery and detection in Spanish}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure