TL;DR
This paper presents a zero-shot approach for multilingual and cross-lingual Word-in-Context (WiC) disambiguation. By leveraging a multilingual transformer and a novel similarity measure between ranked synset lists, the system can disambiguate word senses across languages without requiring language-specific training data, offering a valuable baseline for low-resource scenarios.
What Problem Does This Paper Address?
Word Sense Disambiguation (WSD) is a fundamental task in Natural Language Processing (NLP) aimed at identifying the correct meaning of a word in its given context. Traditional WSD often relies on extensive, sense-annotated corpora, which are labor-intensive to create and scarce, especially for languages other than English. The Word-in-Context (WiC) task was proposed as a simpler surrogate, framing disambiguation as a binary classification problem: do two target words in different contexts have the same meaning? This paper tackles the challenge of extending WiC to multilingual and cross-lingual settings in a zero-shot manner, meaning the system should perform well without explicit training on the target languages or tasks.
Methods
The core of the SzegedAI system is built upon two fine-grained WSD models originally designed for English: LMMS and S-LMMS. These models use transformer encoders (like BERT) to generate contextualized word embeddings. The key innovation for multilingual and cross-lingual capabilities is the replacement of the English-specific BERT encoder with the multilingual XLM-ROBERTa-large model.
Instead of directly predicting a single sense, the system outputs a ranked list of English WordNet synsets for each target word. To determine if two words have the same meaning (WiC task), the system calculates the similarity between these ranked synset lists. Three similarity measures were explored: Jaccard similarity, Ranking-Biased Overlap (RBO), and a harmonic mean of Mean Reciprocal Rank (MRR). A threshold is then determined (using expectation maximization) above which the words are considered to have the same meaning. This approach allows for a more nuanced output than a simple binary classification, providing a ranked list of potential meanings.
Datasets
- Training Data: The underlying fine-grained WSD models were trained on English sense-annotated data, specifically the SemCor dataset and the Princeton WordNet Gloss Corpus (WNGC). The UWA corpus (unambiguous words from Wikipedia and OpenWebTex) was also explored for additional training signal.
- Evaluation Data: The system was evaluated on the SemEval-2021 Task 2 (Multilingual and Cross-lingual Word-in-Context Disambiguation) shared task data. This included multilingual test sets (sentence pairs in the same language: Arabic, English, French, Russian, Chinese) and cross-lingual test sets (English sentence paired with a non-English sentence).
Key Insights and Ideas
- Multilingual Transformers for Zero-Shot WiC: Replacing English-specific encoders with multilingual ones like XLM-ROBERTa is a viable strategy for extending WSD and WiC to new languages without explicit training.
- Ranked Synset Lists as a Richer Output: Providing a ranked list of synsets, rather than just a single prediction, offers a more interpretable and potentially more robust output for similarity comparisons.
- Similarity Measures for Ranked Lists: Different similarity metrics (Jaccard, RBO, MRR-based) can be used to compare ranked synset lists, with Jaccard and RBO showing comparable performance.
- S-LMMS Outperforms LMMS: The S-LMMS model, which incorporates sparsity-inducing dictionary learning, proved more robust to the switch from BERT to XLM-ROBERTa, showing less performance degradation.
- Generic Applicability: The framework's design, particularly its reliance on a multilingual encoder and a language-agnostic similarity measure, makes it easily adaptable to new languages, even those with limited linguistic resources.
Key Results
- The system achieved competitive results on the SemEval-2021 Task 2, though it did not reach the top-scoring submissions.
- Replacing BERT with XLM-ROBERTa led to a performance decrease in fine-grained WSD, but the drop was not severe. S-LMMS was less affected by this change than LMMS.
- Using the 21st layer of the transformer model generally provided a slight edge over using the average of the last four layers for fine-grained WSD.
- For the WiC task, the mS-LMMS approach consistently outperformed mLMMS across both multilingual and cross-lingual settings.
- The choice of similarity calculation (Jaccard/MRR/RBO) had a less significant impact on results compared to the choice of the WSD system (mLMMS vs. mS-LMMS).
Limitations and Future Work
- Performance Gap: The authors acknowledge that their results lag behind top-scoring submissions, suggesting room for improvement.
- No Language-Specific Heuristics: The system did not incorporate language-specific heuristics (like part-of-speech filtering) to maintain its generic nature. This could potentially limit performance on languages where such heuristics are highly effective.
- Future Work: The paper highlights the system's potential as a baseline for WiC evaluations, especially for low-resource languages. Its convenient applicability to new languages without extensive training data is emphasized as a key advantage for future research. The authors suggest that the framework can be applied to virtually any language supported by the multilingual transformer architecture.
Summary updated