TL;DR
This paper explores how well pre-trained multilingual language models (like XLM-RoBERTa, mBERT, and mDistilBERT) can disambiguate word meanings across different languages. They found that fine-tuning these models directly for the task significantly outperforms using them as feature extractors, and XLM-RoBERTa shows a particular strength in cross-lingual scenarios.
What Problem Does This Paper Address?
The core challenge this paper tackles is multilingual and cross-lingual word-in-context (WiC) disambiguation. WiC, in its original form, is about determining if a target word has the same meaning in two different English sentences. This task is crucial for understanding how well language models can grasp nuanced word meanings. The authors extend this to a multilingual setting, where sentences can be in different languages, and the models need to perform this disambiguation without relying on external dictionaries or sense inventories. This is particularly challenging because training data is often limited to a single language (English in this case), forcing models to generalize to unseen languages in a "zero-shot" manner.
Methods
The researchers investigated two primary approaches using three pre-trained multilingual language models:
- Fine-tuning: This involves taking a pre-trained model and training it further on the specific WiC task. They used a "span classification head" that focuses only on the target words within the sentences. This approach directly adapts the model's parameters to the disambiguation task.
- Feature Extraction: Here, the pre-trained models are used to generate contextual embeddings (numerical representations) for the target words. These embeddings are then fed into a separate classifier (either logistic regression or a multi-layer perceptron) to make the disambiguation decision. They also experimented with augmenting these embeddings with dependency syntax information (head and dependent words) to see if it improved performance.
The three models explored were:
- XLM-RoBERTa (XLMR): Trained on a massive dataset across over 100 languages.
- Multilingual BERT (mBERT): Trained on Wikipedia in multiple languages.
- Multilingual Distilled BERT (mDistilBERT): A smaller, faster version of mBERT.
Datasets
The study exclusively used datasets provided by SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC). This dataset extends the original WiC dataset to include five languages: Arabic, Chinese, English, French, and Russian. The task involves binary classification: determining if a word has the same or different meaning in a pair of sentences. Crucially, training data was only available for English-English pairs, making the cross-lingual settings effectively zero-shot.
Key Insights and Ideas
- Fine-tuning reigns supreme: The most significant finding is that fine-tuning the language models directly for the WiC task yields substantially better results than using them as static feature extractors. This suggests that adapting the model's internal representations is more effective than simply leveraging pre-computed embeddings.
- XLMR's cross-lingual prowess: XLM-RoBERTa consistently outperformed mBERT and mDistilBERT in cross-lingual scenarios, both for fine-tuning and feature extraction. This highlights its superior ability to generalize word meanings across different languages.
- mDistilBERT's surprising resilience: While mDistilBERT performed poorly when fine-tuned, it was surprisingly competitive with mBERT and XLMR when used as a feature extractor. This indicates that even a distilled model can provide valuable contextual information.
- Syntax: A mixed bag: Adding dependency syntax information to feature extraction methods yielded mixed results. It improved performance in some cases (e.g., English-Chinese) but not others, suggesting its utility is context-dependent and not a universal boost.
- English-English advantage: As expected, performance was generally stronger on English-English pairs due to the availability of dedicated training data.
Key Results
- Fine-tuned models achieved significantly higher performance than feature extraction models, often by a large margin.
- In the cross-lingual setting, XLM-RoBERTa achieved the best results, outperforming mBERT by 4.1–10.5 percentage points.
- In the multilingual setting, the performance gap between XLM-RoBERTa and mBERT was smaller, with XLM-RoBERTa leading by at most 3.6 percentage points.
- mDistilBERT's fine-tuning performance was notably weak, falling behind by more than 10 percentage points in several cases.
- When used as feature extractors, mDistilBERT was competitive, often performing on par or better than mBERT.
- Using an MLP classifier with feature extraction was generally preferable to logistic regression, leading to large improvements in most cases.
Limitations and Future Work
The authors acknowledge that due to time constraints, they did not use XLM-RoBERTa in all feature extraction systems. They also note that the dependency-based syntax incorporation led to mixed results, suggesting further investigation is needed.
Promising future directions include:
- Investigating XLM-RoBERTa's cross-lingual advantage: Further exploring why XLMR excels in cross-lingual tasks.
- Exploring sub-word model differences: Examining the impact of XLMR's SentencePiece versus mBERT's WordPiece on performance.
- Layer-wise analysis: Investigating the effect of using representations from different layers of the pre-trained models.
Summary updated