TL;DR
This paper explores whether a multilingual Word-in-Context (WiC) model can automate the laborious task of identifying lexical semantic change. While their WiC model achieved competitive results on the LSCDiscovery shared task, it didn't fully replace human annotators, highlighting the nuances of semantic change detection that current models still struggle with.
What Problem Does This Paper Address?
Lexical Semantic Change Discovery (LSCD) aims to track how word meanings evolve over time by analyzing large historical text corpora. This process typically relies on human annotators to meticulously label pairs of word occurrences, determining if they represent the same or different senses. This annotation is time-consuming and expensive, creating a bottleneck for large-scale LSCD research. The authors tackle the challenge of automating this annotation process by leveraging a multilingual Word-in-Context (WiC) model, essentially asking if a machine can perform the same semantic similarity judgments as humans.
Methods
The core of the authors' approach is a WiC model built upon the XLM-R encoder, a powerful multilingual Transformer. This model is trained to classify whether two sentences containing the same target word exhibit the same or different meanings. For the LSCD task, they adapt this WiC model in several ways:
- WiC Training: They experiment with different fine-tuning strategies, including training on multilingual datasets (MCL-WiC), Russian-specific datasets (RuSemShift), and Spanish datasets (DWUG_es, XL-WSD).
- APD (Average Pairwise Distance): For the graded change discovery subtask, they use the WiC model to score pairs of sentences. The average of these scores (predicted probabilities of different senses) for a given word is used as its "graded change" score.
- Binary Classification: For binary subtasks (change vs. no change, sense gain, sense loss), they employ thresholding on the APD scores. They also explore two binarization strategies (bin1 and bin2) for the training data, which differ in how they handle intermediate similarity scores.
- Correlation Clustering (CC): As an alternative to APD, they also experiment with a clustering-based approach, treating WiC predictions as edges in a graph to cluster word uses.
Datasets
- MCL-WiC: A multilingual and cross-lingual Word-in-Context dataset used for initial training.
- RuSemShift: A dataset for historical lexical semantic change in Russian, used for fine-tuning.
- DWUG_es: The development set for the LSCDiscovery shared task in Spanish.
- XL-WSD: A large, cross-lingual Word Sense Disambiguation dataset used to generate additional Spanish training data.
- LSCDiscovery Test Set: The primary evaluation benchmark for the shared task, consisting of Spanish words and their historical corpora.
Key Insights and Ideas
- WiC as a Proxy for Human Annotation: The paper demonstrates that WiC models can indeed serve as a reasonable proxy for human semantic similarity judgments, achieving competitive results on LSCD tasks.
- Data Augmentation is Crucial: Fine-tuning on additional Spanish data, particularly from the XL-WSD dataset, significantly boosted performance, suggesting that diverse and relevant training data is key.
- Binarization Matters: The choice of how to binarize human annotations for WiC training (e.g., bin1 vs. bin2) had a noticeable impact on performance, with filtering out intermediate scores (bin2) often leading to better results.
- Multi-step vs. Single-step Fine-tuning: The authors explore different fine-tuning sequences, finding that while multi-step fine-tuning can be beneficial, a single-step approach on all data also shows promise and might prevent forgetting.
- APD vs. CC: The APD-based method generally outperformed the Correlation Clustering approach for the binary subtasks, suggesting that direct probability-based scoring might be more robust in this context.
Key Results
- Graded Change Discovery: The authors' best submission achieved the 2nd best result in the graded change discovery subtask.
- Binary Change Detection: Their F1-score was 0.655, placing them 5th, slightly below the best submission's 0.716.
- Sense Gain Detection: They outperformed all other participants in the optional sense gain detection subtask.
- Post-evaluation Improvements: Changing binarization to "bin2" and setting a natural threshold of 0.5 improved binary change and sense loss detection results to be comparable with the 2nd best leaderboard submission.
- Computational Cost: Processing approximately 388K sentence pairs for the graded change task took about 3 hours on a V100 GPU.
Limitations and Future Work
- Not a Full Replacement: The paper concludes that while their WiC model is promising, it's not yet good enough to fully replace human annotators for LSCD.
- Suboptimal Binarization: The authors suggest that suboptimal binarization of Spanish data might have limited further improvements.
- Correlation Clustering Analysis: A deeper analysis is needed to understand why the CC method performed worse than APD.
- Future Directions: Experimenting with different fine-tuning strategies (multi-step vs. single-step) and further improving the WiC model's ability to reproduce human judgments are highlighted as promising avenues for future work. Investigating the impact of sampling more examples for real-world applications is also mentioned.
Summary updated