SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC)

Federico Martelli, Najla Kalach, Gabriele Tola, Roberto NavigliACL 2021

TL;DR

This paper introduces MCL-WiC, the first task and manually curated dataset for multilingual and cross-lingual Word-in-Context (WiC) disambiguation. It addresses the limitations of existing WiC benchmarks by enabling evaluation across multiple languages and parts of speech, crucial for developing truly robust NLP systems.

What Problem Does This Paper Address?

Natural Language Processing (NLP) models have made significant strides in understanding language, but a core challenge remains: accurately discerning the meaning of a word based on its context. Traditional Word Sense Disambiguation (WSD) tasks often rely on predefined sense inventories (like WordNet), which are not always available for all languages and can limit the evaluation of modern contextualized word embeddings. Existing WiC benchmarks, while dropping the need for fixed inventories, are often automatically generated, limited to specific languages (primarily English) and parts of speech (nouns and verbs), and don't fully support cross-lingual evaluation. This gap hinders the development of NLP systems that can generalize effectively across diverse linguistic contexts.

Methods

The paper introduces the Multilingual and Cross-lingual Word-in-Context (MCL-WiC) task, framed as a binary classification problem. Systems must determine if two target words, each in a different sentence, share the same meaning.

The task is split into two sub-tasks:

  • Multilingual: Target words and their contexts are in the same language.
  • Cross-lingual: Target words and their contexts are in different languages (specifically, English paired with Arabic, Chinese, French, or Russian).

A key methodological contribution is the creation of a manually-annotated dataset for MCL-WiC, covering five languages: Arabic, Chinese, English, French, and Russian. This manual curation ensures higher quality and addresses the limitations of automatically generated datasets.

Datasets

The core contribution is the MCL-WiC dataset, a manually curated benchmark for multilingual and cross-lingual WiC.

  • Languages: Arabic (Ar), Chinese (Zh), English (En), French (Fr), and Russian (Ru).
  • Sub-tasks:
    • Multilingual: Ar-Ar, En-En, Fr-Fr, Ru-Ru, Zh-Zh.
  • Cross-lingual: En-Ar, En-Fr, En-Ru, En-Zh.
  • Data Sources: Leveraged BabelNet for lexemes, and corpora like the United Nations Parallel Corpus (UNPC) and Wikipedia for sentence pairs.
  • Annotation: Performed by eight annotators with linguistic proficiency, using multiple dictionaries for justification and aiming for lexicographic motivation to reduce subjectivity.
  • Comparison to Existing Datasets: Unlike automatically retrieved datasets like WiC and XL-WiC, MCL-WiC is entirely manually created. It also expands coverage to more languages and parts of speech, and crucially, enables cross-lingual evaluation scenarios where systems are tested simultaneously in two languages.

Key Insights and Ideas

  • The Need for Manual Curation: The paper strongly advocates for manual annotation in WiC tasks, arguing it significantly reduces subjectivity and leads to more adequate and consistent data compared to automatically retrieved datasets.
  • Cross-Lingual Evaluation is Crucial: The introduction of cross-lingual WiC scenarios is a significant step, allowing researchers to assess how well models transfer knowledge of word meanings across languages, a critical aspect of true language understanding.
  • Language-Specific Challenges: The paper highlights unique linguistic issues for Arabic (omission of vocalization), Chinese (character-based ambiguity and polysyllabic words), and Russian (verbal aspects like perfective/imperfective verbs), underscoring the complexity of building universal NLP models.
  • XLM-ROBERTa Dominance: The results show that XLM-ROBERTa consistently outperformed BERT across most language combinations, highlighting the effectiveness of large, multilingual pre-trained models for this task.

Key Results

  • High Performance Achieved: Systems achieved high accuracy (85-93%) on the MCL-WiC dataset, suggesting that current models are capable of capturing sense distinctions without relying on fixed sense inventories.
  • Significant Improvement over Baselines: The best-performing systems showed absolute improvements of up to 27.1 points over the provided baselines.
  • XLM-ROBERTa Outperforms BERT: XLM-ROBERTa was the preferred model for most participants, demonstrating its superiority for multilingual and cross-lingual tasks.
  • En-En Performance: The highest score (93.3% accuracy) was achieved on the En-En dataset.
  • Chinese as Second Best: Chinese performed notably well in both monolingual (Zh-Zh) and cross-lingual (En-Zh) settings.
  • Arabic as Most Difficult: Arabic presented the biggest challenge, particularly in the Ar-Ar sub-task, possibly due to its morpho-semantic features.
  • Zero-Shot Advantage: Zero-shot approaches in the cross-lingual sub-task performed slightly better, likely due to the presence of English in both training and testing, and the availability of English WiC-style datasets.
  • Data Augmentation Benefits: Swapping sentences within pairs and adding signals (like special markings) around target words consistently improved performance.

Limitations and Future Work

  • Limitations:
    • The paper notes that while the dataset covers open-class parts of speech, it avoids phrasal verbs and multi-word expressions.
  • The cross-lingual sub-task primarily uses English as the source language.
  • Future Work:
    • The authors suggest exploring the integration of multi-word expressions and idiomatic phrases into the WiC task to evaluate systems' ability to correctly discriminate the semantics of these complex linguistic constructs, especially non-compositional ones.

Summary updated

BibTeX
@inproceedings{martelli2021semeval,
  author = {Federico Martelli and Najla Kalach and Gabriele Tola and Roberto Navigli},
  title = {{SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC)}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure