TL;DR
This paper introduces two novel systems for detecting lexical semantic change (LSC) in Spanish. One system frames binary LSC detection as a word sense disambiguation (WSD) problem, achieving competitive results. The other system builds upon a previous state-of-the-art approach for graded LSC discovery, incorporating XLM-ROBERTa for improved contextual embeddings.
What Problem Does This Paper Address?
Lexical semantic change (LSC) refers to how the meanings of words evolve over time. This is a crucial area for understanding language evolution and has applications in fields like historical linguistics and natural language processing. The LSCDiscovery shared task in Spanish presents two challenges: 1) graded change discovery, where words are ranked by their degree of semantic change between two time periods, and 2) binary change detection, which determines if a word has undergone any significant semantic change. Existing methods often rely on complex embeddings and may not be easily interpretable or multilingual. This paper aims to develop effective and potentially more interpretable systems for both LSC tasks in Spanish.
Methods
The authors propose distinct approaches for the two LSC tasks:
- Binary Change Detection (Phase 2): This system frames the problem as Word Sense Disambiguation (WSD). They leverage the AMUSE WSD system, which uses multilingual XLM-ROBERTa embeddings. For each target word, they compute sense frequency distributions in both old and modern corpora. A change is detected if a sense appears in one corpus but not the other, or if the relative change in a sense's probability exceeds a tuned threshold. This approach offers interpretability by identifying specific senses that emerge or disappear.
- Graded Change Discovery (Phase 1): This system builds upon the CIRCE system, a previous winner in a similar task. They use a combination of static (Skip-Gram) and contextual (XLM-ROBERTa) embeddings. A key modification is training the XLM-ROBERTa model to predict randomly masked words rather than the time period, which they hypothesize better aligns with predicting word meaning. They then extract embeddings from the masked token's representation and compute pairwise Euclidean distances. The final ranking is a weighted combination of rankings derived from static and contextual embeddings, with weights tuned on the development set.
Datasets
The paper uses datasets from the LSCDiscovery shared task on Spanish. These datasets consist of diachronic corpus pairs (old and modern texts) and annotated target words with their usages. The gold labels for the shared task were derived from manual annotations and sense frequency distributions.
Key Insights and Ideas
- WSD as a powerful tool for LSC: Framing binary LSC detection as WSD provides an interpretable way to understand semantic change by focusing on specific sense shifts rather than just overall distributional changes.
- XLM-ROBERTa for better contextual embeddings: Using XLM-ROBERTa, a multilingual model, and training it for masked word prediction (instead of time period prediction) proved beneficial for capturing contextual semantic nuances in the graded change task.
- Multilingual capabilities: The use of XLM-ROBERTa and the AMUSE WSD system inherently provides multilingual capabilities, addressing a potential limitation of previous LSC methods that might require language-specific resources.
- Static embeddings still relevant: Despite the rise of contextual embeddings, static embeddings still contribute valuable information for LSC detection, and combining them with contextual embeddings yields better results.
Key Results
- Phase 1 (Graded Change): The system achieved a Spearman's rank-order correlation of 0.5731 on the evaluation set, placing it among the top-performing systems. This is a significant improvement over the baseline CIRCE system's correlation of 0.1894.
- Phase 2 (Binary Change): The WSD-based system achieved an F-score of 71% on the evaluation set, ranking it second in the competition. For the optional tasks of sense gain and loss detection, their results placed third and second, respectively.
Limitations and Future Work
- Computational Cost: Training the XLM-ROBERTa model for contextual embeddings is computationally expensive, taking several hours on powerful hardware. Static embeddings are significantly faster.
- Spurious Sense Identification: The WSD approach can sometimes be affected by incorrect Part-of-Speech (POS) tags assigned by the WSD system, leading to spurious new sense identifications. Experimenting with POS tag matching improved results slightly but was not the main method.
- Future Work: The authors suggest exploring combinations of more than two models for graded LSC discovery. They also propose investigating whether their methods for one task could be adapted to the other subtask.
Summary updated