TransWiC at SemEval-2021 Task 2: Transformer-based Multilingual and Cross-lingual Word-in-Context Disambiguation

Hansi Hettiarachchi, Tharindu RanasingheACL 2021

TL;DR

This paper tackles the challenge of Word-in-Context (WiC) disambiguation across multiple languages, a crucial task for NLP applications. The authors propose a novel approach using transformer models without relying on language-specific resources, achieving competitive results on the SemEval-2021 Task 2, particularly with their "<B> + [CLS]" strategy.

What Problem Does This Paper Address?

Understanding the meaning of a word in its specific context is fundamental for many Natural Language Processing (NLP) tasks, such as question answering, summarization, and information retrieval. However, words often have multiple meanings (polysemy), making it difficult to determine the correct sense. Traditional Word Sense Disambiguation (WSD) methods often rely on external knowledge bases like WordNet or BabelNet, which are typically limited to high-resource languages, hindering their application in multilingual and cross-lingual settings. SemEval-2021 Task 2 specifically addresses this gap by focusing on a binary classification problem: determining if a target word has the same or different meaning in two given contexts, across both monolingual and cross-lingual scenarios, without using predefined sense inventories.

Methods

The core of the TransWiC approach lies in leveraging pre-trained transformer models, specifically BERT and XLM-R, to handle the WiC task. Instead of relying on language-specific resources, the authors focus on adapting the standard sentence-pair classification architecture. They introduce several "strategies" that modify how the input sentences and target words are processed and fed into the transformer.

Key innovations include:

  • Preprocessing: Special tokens <B> (beginning) and <E> (end) are added around the target word in both sentences to explicitly mark its position.
  • Diverse Strategies: The paper explores various ways to combine the transformer's output, including:
  • Using the special [CLS] token's representation.
  • Focusing on the representations of the marked target words (<B> and <E> tokens).
  • Employing pooling strategies to create fixed-length embeddings for target words that might be split into sub-word units.
  • Combining these approaches with the [CLS] token.

The authors found that the "<B> + [CLS]" strategy, which concatenates the representations of the marked target words and the [CLS] token, performed particularly well. For languages with limited training data, they employed few-shot learning by initializing models with weights from the English-English task. For cross-lingual tasks with no training data, they utilized zero-shot learning, capitalizing on the inherent cross-lingual transfer capabilities of multilingual transformers.

Datasets

The paper utilizes the datasets provided for SemEval-2021 Task 2. These datasets are designed for a binary classification task and cover five languages: English, Arabic, French, Russian, and Chinese.

  • Monolingual: Sentence pairs within the same language (e.g., English-English, Arabic-Arabic).
  • Cross-lingual: Sentence pairs in different languages (e.g., English-Arabic, English-French).

For training, only the English-English monolingual data was available (8,000 instances). Development sets (1,000 instances per language) and test sets (1,000 instances per language pair) were provided for both monolingual and cross-lingual settings. Notably, no training or development data was provided for the cross-lingual tasks.

Key Insights and Ideas

  • Transformer Powerhouse: Pre-trained transformer models, especially multilingual ones like XLM-R, are highly effective for WiC tasks, even without task-specific fine-tuning on every language.
  • Strategic Input Matters: The way the target word and its context are presented to the transformer significantly impacts performance. The "<B> + [CLS]" strategy, which explicitly marks the target word and leverages the global [CLS] token, proved to be a winning combination.
  • Cross-lingual Transfer is Key: The success of multilingual transformers in zero-shot settings for cross-lingual WiC highlights their ability to generalize knowledge across languages, a critical feature for low-resource scenarios.
  • No Language-Specific Resources Needed: The paper demonstrates that competitive results can be achieved without relying on external linguistic resources or language-specific preprocessing, simplifying the pipeline and increasing applicability.

Key Results

The paper reports accuracy scores on the English-English development set and the test sets for all language pairs.

  • English-English: The best performing strategy, "<B> + [CLS]", achieved an accuracy of 0.8810 with XLM-R and 0.8590 with BERT. This is highly competitive with the best overall system for this subtask (0.93 accuracy).
  • Monolingual and Cross-lingual Test Sets: Table 3 summarizes the test set accuracies. The "<B> + [CLS]" strategy achieved 0.9040 for English-English and competitive scores across other monolingual and cross-lingual pairs, such as 0.7800 for Arabic-Arabic and 0.6900 for English-Russian.
  • XLM-R Outperforms BERT: Surprisingly, for this task, the XLM-R models generally outperformed BERT-large models, even though they are typically language-specific models that are expected to perform better.

Limitations and Future Work

  • Cross-lingual Performance Improvement: While the cross-lingual results are satisfactory given the zero-shot approach, the authors believe there is room for improvement.
  • Exploring Language-Specific Resources: The paper acknowledges that incorporating language-specific processing or resources might further enhance performance, a direction they plan to explore.
  • Alternative Architectures: The authors are interested in experimenting with other neural network architectures, such as Siamese transformer networks, which have shown promise in sentence-pair classification tasks.
  • Multi-task Learning: Investigating whether transfer learning from related tasks, like semantic textual similarity, could benefit the WiC task is another avenue for future research.

Summary updated

BibTeX
@inproceedings{hettiarachchi2021transwic,
  author = {Hansi Hettiarachchi and Tharindu Ranasinghe},
  title = {{TransWiC at SemEval-2021 Task 2: Transformer-based Multilingual and Cross-lingual Word-in-Context Disambiguation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure