Abstract
Capturing word meaning in context and distinguishing between correspondences and variations across languages is key to building successful multilingual and cross-lingual text representation models. However, existing multilingual evaluation datasets that evaluate lexical semantics "in-context" have various limitations. In particular, 1) their language coverage is restricted to high-resource languages and skewed in favor of only a few language families and areas, 2) a design that makes the task solvable via superficial cues, which results in artificially inflated (and sometimes super-human) performances of pretrained encoders, on many target languages, which limits their usefulness for model probing and diagnostics, and 3) little support for cross-lingual evaluation. In order to address these gaps, we present AM2iCo (Adversarial and Multilingual Meaning in Context), a wide-coverage cross-lingual and multilingual evaluation set; it aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts for 14 language pairs. We conduct a series of experiments in a wide range of setups and demonstrate the challenging nature of AM2iCo. The results reveal that current SotA pretrained encoders substantially lag behind human performance, and the largest gaps are observed for low-resource languages and languages dissimilar to English.
Introduction
In natural language processing, the Word-in-Context (WiC) task evaluates whether a specific word has the same meaning across different sentences. While models like BERT and XLM-R have reported "super-human" performance on previous WiC benchmarks, these results are often inflated due to superficial cues. Existing datasets typically focus on high-resource Indo-European languages and lack robust cross-lingual testing. To address this, researchers introduced AM²iCo (Adversarial and Multilingual Meaning in Context), a benchmark designed to stress-test models across 15 diverse languages using adversarial examples that prevent models from relying on simple pattern matching.
Methods and Adversarial Design
AM²iCo leverages Wikipedia's interlinked, disambiguated text to identify cross-lingual concept correspondences. To ensure the task is not solvable via shortcuts, the researchers implemented two key strategies:
- Data Selection Heuristics: Filtering monosemous words and ensuring negative examples are not just subtle variations of the same concept.
- Adversarial Examples: The "secret sauce" involves replacing a target word with a related word (e.g., replacing "Apollo" with "NASA") in the same context. This forces the model to understand the semantic interplay between the word and its surrounding text, rather than relying on simple word-level pattern matching.
Dataset Characteristics
The AM²iCo dataset spans 15 languages and 10 language families, including low-resource languages like Basque, Georgian, Bengali, Kazakh, and Urdu. Unlike previous benchmarks, AM²iCo provides significantly longer context lengths (averaging 53.5 words vs. 17-26 in older sets) and a more balanced representation of non-Western languages.
Key Results and Insights
Experimental results highlight a significant gap between human and machine performance. While humans achieve ~90.6% accuracy, state-of-the-art models like mBERT and XLM-R struggle, particularly on adversarial examples.
- Adversarial Impact: Model performance drops sharply when "shortcuts" are blocked. For instance, mBERT's accuracy on German dropped from 84.2% to 73.7% when adversarial examples were introduced.
- Cross-Lingual Transfer: For low-resource languages, transferring knowledge from a linguistically related source (e.g., Russian to Kazakh) is more effective than using a high-resource but unrelated source (e.g., German), even if the scripts differ.
- Joint Training: Training a single model on all language pairs simultaneously (Joint Multilingual Learning) yields the best overall performance, particularly benefiting the lowest-resource languages.
Conclusion
AM²iCo serves as a rigorous diagnostic tool for multilingual models. The findings suggest that current models are surprisingly poor at "contextualization"—the ability to let surrounding words shift the meaning of a target word. Future work should expand this benchmark to more informal text domains and move beyond binary classification to capture the nuanced spectrum of human language meaning.
Summary updated