AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples

Qianchu Liu, Edoardo Maria Ponti, Diana McCarthy, Ivan Vulic, Anna Korhonen

TL;DR

This paper introduces AM²ICO, a new benchmark for evaluating how well language models understand word meaning in context across many languages, especially low-resource ones. They found that current state-of-the-art models still lag significantly behind human performance, highlighting the need for better cross-lingual understanding.

What Problem Does This Paper Address?

Understanding word meaning in context is crucial for building effective language technologies. While there are datasets for evaluating this, they often focus on high-resource languages, can be solved with superficial cues, and don't adequately support cross-lingual evaluation. This limits our ability to assess and improve multilingual language models, especially for the vast majority of the world's languages that are not high-resource. The authors aim to bridge this gap by creating a more comprehensive and challenging evaluation resource.

Methods

The core of this work is the creation of the AM²ICO dataset. They leverage Wikipedia's cross-lingual links to identify corresponding concepts across languages. For each concept, they extract sentence contexts and then construct pairs of contexts (one in English, one in a target language) where a word is used. To ensure the task is challenging and not solvable by superficial cues, they employ several strategies:

  • Adversarial Examples: They create negative examples where a related word (e.g., "NASA" instead of "Apollo" when discussing spaceships) is used, requiring a deeper understanding of semantic nuances.
  • Data Selection Heuristics: They filter out ambiguous links and overly fine-grained distinctions to ensure that examples truly test contextual understanding.
  • Cross-lingual Evaluation: The dataset is designed to directly support evaluating how well models transfer knowledge between languages.

Datasets

The primary contribution is the AM²ICO dataset, which covers 15 languages and 14 language pairs (English paired with 14 target languages). This dataset is built using Wikipedia and includes adversarial examples. It aims to be more typologically diverse and cover a wider vocabulary than existing benchmarks like WiC and XL-WiC.

Key Insights and Ideas

  • Current Models Struggle with Nuance: The inclusion of adversarial examples in AM²ICO significantly increases the difficulty for models, revealing that they often rely on superficial cues rather than deep semantic understanding.
  • Low-Resource and Distant Languages are a Major Hurdle: Models perform substantially worse on languages that are low-resource or typologically dissimilar to English, indicating a significant gap in cross-lingual transfer capabilities.
  • Wikipedia is a Rich Source for Cross-Lingual Semantics: The paper demonstrates the effectiveness of using Wikipedia's cross-lingual links as a foundation for creating large-scale, multilingual semantic evaluation resources.
  • Fine-tuning vs. Metric-Based Evaluation: While metric-based approaches provide a baseline, fine-tuning models on AM²ICO shows a greater sensitivity to data size, especially for low-resource languages.

Key Results

  • Human Performance Benchmark: Human accuracy on AM²ICO is high (around 90%), while current state-of-the-art models (MBERT, XLM-R) achieve significantly lower scores, with gaps of up to 13.5-16.5 points on specific language pairs when adversarial examples are included.
  • Supervised Learning: In a supervised setting, MBERT and XLM-R achieve accuracies ranging from 55.5% to 65.0% across languages.
  • Zero-Shot Transfer: Performance varies, but generally, models struggle to transfer knowledge effectively, especially to distant languages. For instance, German as a source language yields an average score of around 71-72% for MBERT and XLM-R, but this drops significantly for target languages like ZH or AR.
  • Joint Multilingual Learning: Training on multiple languages simultaneously provides a substantial boost, improving scores for low-resource languages by up to 6.3 points, though it doesn't fully close the gap with high-resource languages.

Limitations and Future Work

  • Resource Constraints: The authors note that experiments with larger model variants were limited due to resource constraints.
  • Model Size: The current evaluation focuses on specific multilingual models (MBERT, XLM-R); future work could explore other architectures.
  • Language Coverage: While AM²ICO is more diverse than previous datasets, it still covers only 15 languages. Expanding to even more typologically diverse languages would be beneficial.
  • Further Exploration of Transfer: The paper suggests further research into understanding the impact of language relatedness and exploring more sophisticated cross-lingual transfer strategies. The authors hope AM²ICO will guide future research in effective representation learning across languages.

Summary updated

BibTeX
@inproceedings{liu2021am,
  author = {Qianchu Liu and Edoardo Maria Ponti and Diana McCarthy and Ivan Vulic and Anna Korhonen},
  title = {{AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.571},
}
Figure