Introduction
Word Sense Disambiguation (WSD) is a fundamental task in Natural Language Processing (NLP) that involves identifying the correct meaning of a word based on its context. Despite significant progress driven by Transformer-based models, WSD research has remained largely English-centric. This "language bottleneck" stems from a lack of high-quality, unified datasets for languages other than English, making it difficult to evaluate and compare cross-lingual systems.
The XL-WSD Framework
To address these challenges, the authors introduce XL-WSD, a comprehensive framework designed to push WSD into a truly multilingual era. The framework is built on three pillars:
- Unified Sense Inventory: By mapping all senses to BabelNet 4.0 (linked to WordNet 3.0), the framework ensures semantic consistency across 18 different languages.
- Silver Data Generation: To overcome the scarcity of human-annotated data, the researchers employed a "translate-and-project" method, creating large-scale synthetic training sets for 15 target languages.
- Standardized Evaluation: XL-WSD unifies legacy datasets and introduces over 70,000 new gold annotations, providing a robust benchmark for 18 languages across six linguistic families.
Experimental Setup and Results
The study evaluated several state-of-the-art models, including XLM-RoBERTa (XLMR) and BERT, across various settings: zero-shot (training on English only), language-specific (training on silver data), and traditional knowledge-based approaches.
The results highlight the remarkable capability of large-scale multilingual models. XLMR-Large emerged as the strongest baseline, achieving an average F1 score of 65.66 across all non-English languages in a zero-shot setting.
Key Insights
- Zero-Shot Superiority: Surprisingly, zero-shot transfer from English often outperformed models trained on language-specific "silver" data. This suggests that the conceptual representations in models like XLMR are highly robust across languages.
- The Language Gap: While English performance reaches F1 scores in the high 70s, many languages still lag behind. Basque (47.15) and Catalan (49.97) represent significant challenges for current models.
- Part-of-Speech Difficulty: Disambiguating verbs remains significantly harder than nouns, with F1 scores typically 10 points lower for verbs across the board.
- The Long Tail: Performance correlates strongly with sense frequency; rare senses in the "long tail" of the distribution remain difficult for neural models to capture accurately.
Conclusion
XL-WSD provides a much-needed standardized platform for multilingual WSD research. While neural models show impressive zero-shot capabilities, the significant performance gap between English and other languages—as well as the difficulty with verbs and rare senses—indicates that the field is far from solved. Future work will focus on expanding gold annotations for low-resource languages and improving model robustness for infrequent meanings.
The code and evaluation suite are available at https://sapienzanlp.github.io/xl-wsd/.
Summary updated