XL-WSD: An Extra-Large and Cross-Lingual Evaluation Framework for Word Sense Disambiguation

Tommaso Pasini, Alessandro Raganato, Roberto NavigliAAAI 2021 · DOI

Introduction

Word Sense Disambiguation (WSD) is a fundamental task in Natural Language Processing (NLP) that involves identifying the correct meaning of a word based on its context. Despite significant progress driven by Transformer-based models, WSD research has remained largely English-centric. This "language bottleneck" stems from a lack of high-quality, unified datasets for languages other than English, making it difficult to evaluate and compare cross-lingual systems.

The XL-WSD Framework

To address these challenges, the authors introduce XL-WSD, a comprehensive framework designed to push WSD into a truly multilingual era. The framework is built on three pillars:

  1. Unified Sense Inventory: By mapping all senses to BabelNet 4.0 (linked to WordNet 3.0), the framework ensures semantic consistency across 18 different languages.
  2. Silver Data Generation: To overcome the scarcity of human-annotated data, the researchers employed a "translate-and-project" method, creating large-scale synthetic training sets for 15 target languages.
  3. Standardized Evaluation: XL-WSD unifies legacy datasets and introduces over 70,000 new gold annotations, providing a robust benchmark for 18 languages across six linguistic families.
Table 1: Statistics of the XL-WSD dataset across 18 languages, showing word types, polysemy, and instance counts for training and testing.

Experimental Setup and Results

The study evaluated several state-of-the-art models, including XLM-RoBERTa (XLMR) and BERT, across various settings: zero-shot (training on English only), language-specific (training on silver data), and traditional knowledge-based approaches.

The results highlight the remarkable capability of large-scale multilingual models. XLMR-Large emerged as the strongest baseline, achieving an average F1 score of 65.66 across all non-English languages in a zero-shot setting.

Table 2: Performance comparison (F1 score) of neural and knowledge-based models across 18 languages in zero-shot and language-specific settings.

Key Insights

  • Zero-Shot Superiority: Surprisingly, zero-shot transfer from English often outperformed models trained on language-specific "silver" data. This suggests that the conceptual representations in models like XLMR are highly robust across languages.
  • The Language Gap: While English performance reaches F1 scores in the high 70s, many languages still lag behind. Basque (47.15) and Catalan (49.97) represent significant challenges for current models.
  • Part-of-Speech Difficulty: Disambiguating verbs remains significantly harder than nouns, with F1 scores typically 10 points lower for verbs across the board.
  • The Long Tail: Performance correlates strongly with sense frequency; rare senses in the "long tail" of the distribution remain difficult for neural models to capture accurately.

Conclusion

XL-WSD provides a much-needed standardized platform for multilingual WSD research. While neural models show impressive zero-shot capabilities, the significant performance gap between English and other languages—as well as the difficulty with verbs and rare senses—indicates that the field is far from solved. Future work will focus on expanding gold annotations for low-resource languages and improving model robustness for infrequent meanings.

The code and evaluation suite are available at https://sapienzanlp.github.io/xl-wsd/.

Summary updated

BibTeX
@inproceedings{pasini2021xl,
  author = {Tommaso Pasini and Alessandro Raganato and Roberto Navigli},
  title = {{XL-WSD: An Extra-Large and Cross-Lingual Evaluation Framework for Word Sense Disambiguation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i15.17609},
}
Figure