TL;DR
This paper presents two approaches for the Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC) task: a question-answering (QA) formulation and a binary classification (BC) approach. The BC approach, particularly an ensemble of XLM-R models with specific aggregation and pooling strategies, achieved top rankings in the multilingual subtask and strong performance in the cross-lingual subtask, demonstrating the effectiveness of fine-tuned large language models for this challenging task.
What Problem Does This Paper Address?
The paper tackles the Word-in-Context Disambiguation (WIC) task, a crucial sub-problem within Natural Language Processing (NLP) that aims to determine if a target word in two different sentences has the same or a different meaning. This is particularly challenging in multilingual and cross-lingual settings, where languages may have different semantic nuances and word senses. The SemEval-2021 Task 2 specifically focused on this by asking participants to classify sentence pairs, where sentences could be in the same language (multilingual) or in different languages (cross-lingual). The core challenge is to equip models with the ability to understand and compare word meanings across diverse linguistic contexts without relying on predefined sense inventories, which are often language-specific and incomplete.
Methods
The authors explored two primary methods:
- Question Answering (QA) System: This approach reframed the WIC task as a span prediction problem. The model was trained to identify the span of a word with the same meaning as the target word in the first sentence, within the second sentence. It also predicted the part-of-speech (PoS) of the target word. The input to the XLM-R model was carefully constructed to include both sentences and the target word, marked with special tokens.
- Binary Classification (BC) System: This more traditional approach fine-tuned XLM-R as a binary classifier. Given two sentences and a target word, the model predicted whether the word's meaning was the same or different. Key innovations in this approach included:
- Data Augmentation: Flipping the order of sentences in training examples to double the dataset size.
- Subword Pooling: Experimenting with different methods (mean, max, first pooling, and their combinations) to aggregate contextualized embeddings of the target word's subwords. Mean pooling was found to be the most effective.
- Target Aggregation: Combining contextualized embeddings using difference, multiplication, and concatenation. The combination of difference (non-normalized) and component-wise product (normalized) proved to be superior.
- Batch Normalization: Applying batch normalization before the classification head significantly improved performance on non-English languages.
- Ensembling: Combining multiple trained models to achieve better overall performance.
Datasets
The paper utilized the datasets provided for the SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation. These datasets included sentence pairs for training, development, and testing across Arabic, Chinese, English, French, and Russian languages, covering both monolingual (e.g., en-en, ar-ar) and cross-lingual (e.g., en-ar, en-fr) scenarios.
Key Insights and Ideas
- Binary Classification Outperforms QA: The authors found that the binary classification approach, despite being more traditional, yielded significantly better results than the question-answering formulation for the WIC task.
- Mean Pooling is King: For aggregating subword embeddings of the target word, mean pooling consistently outperformed other methods, especially for non-English languages where subword tokenization is more frequent.
- Smart Aggregation Beats Simple Concatenation: Combining contextualized embeddings through difference and component-wise product (the "dmn" technique) was substantially more effective than simple concatenation, highlighting the importance of how contextual information is fused.
- Batch Normalization Boosts Multilingual Performance: While batch normalization had a modest impact on English performance, it provided a significant boost for other languages, likely due to the larger proportion of English data in the training set.
- Data Augmentation is Generally Beneficial: Augmenting the training data by swapping sentence order did not decrease performance and was consistently applied in subsequent experiments.
- Multilingual Training Data is Crucial: Including non-English training data improved performance across the board, even for English-only scenarios, demonstrating the benefits of cross-lingual transfer learning.
Key Results
The paper reports strong performance for their binary classification systems. Their ensemble of XLM-R models achieved:
- 2nd place for the monolingual subtask in Arabic.
- 3rd place for the monolingual subtask in French and Russian.
- 6th place in the cross-lingual subtask for French and Arabic.
In post-evaluation experiments, their refined XLM-R.large models, using the "dmn" aggregation and mean pooling, further improved results, leading to better rankings in several multilingual and cross-lingual test sets. For instance, they improved their rank in test.ru-ru from 3rd to 2nd, test.fr-fr from 2nd to 1st, and test.en-ar from 6th to 4th.
Limitations and Future Work
- QA System Underperformance: The QA system, while an interesting formulation, did not achieve competitive results compared to the BC approach.
- Subword Pooling Combinations: While combinations of subword pooling methods were explored, they did not yield improvements over single mean pooling.
- Future Work: The authors suggest further exploration of different aggregation techniques and pooling strategies. They also implicitly point towards the continued effectiveness of large pre-trained models like XLM-R for complex NLP tasks. The paper also highlights the benefit of multilingual training data, suggesting that further leveraging such data could lead to even greater gains.
Summary updated