GX at SemEval-2021 Task 2: BERT with Lemma Information for MCL-WiC Task

Wanying XieACL 2021

TL;DR

This paper introduces the GX system for the Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC) task. The system leverages BERT's contextual embeddings and explores novel ways to incorporate lemma information, achieving strong results on English and Chinese tasks, particularly by addressing the challenge of limited data for non-English languages through neural machine translation.

What Problem Does This Paper Address?

The core challenge tackled is Word-in-Context Disambiguation (WiC), specifically in a multilingual and cross-lingual setting (MCL-WiC). WiC aims to determine if a target word has the same meaning across two different contexts. Traditional approaches often rely on fixed sense inventories, which are difficult to manage across many languages. The MCL-WiC task, introduced in SemEval-2021, broadens this to include multiple languages and cross-lingual comparisons, making it significantly more complex. A key hurdle is the scarcity of labeled data for many languages, which hinders the direct application of powerful pre-trained models like BERT.

Methods

The GX system primarily utilizes BERT (Bidirectional Encoder Representations from Transformers) as its foundation. BERT's strength lies in its ability to generate context-specific word embeddings by jointly considering left and right contexts. The authors fine-tune a pre-trained BERT model on the provided English training data.

For languages lacking training data, such as Chinese, they employ a clever strategy: Neural Machine Translation (NMT). They train an English-to-Chinese NMT model (based on the Transformer architecture) and use it to translate the English training data into Chinese. This "pseudo-data" is then used to fine-tune a Chinese BERT model.

Beyond standard BERT fine-tuning, the paper explores several novel input modifications to enhance performance:

  • Lemma Token: Explicitly adding the target word's lemma to the input sequence.
  • Segment Embedding ($E_c$): Modifying the segment embedding for the lemma token to emphasize its importance.
  • Lemma Embeddings: Introducing specialized embeddings to distinguish lemma tokens from regular tokens, further highlighting their role.

Datasets

The primary dataset used is the MCL-WiC dataset released for SemEval-2021 Task 2. This dataset includes both multilingual (within-language) and cross-lingual (across-language) instances. For the Chinese setting, the English portion of the MCL-WiC training data was translated using an NMT model trained on the UNPC v1.0 and MultiUN v1 datasets (part of WMT17).

Key Insights and Ideas

  • BERT is a powerful baseline: Even standard fine-tuning of BERT yields good results for WiC tasks.
  • Lemma information matters: Explicitly incorporating the target word's lemma (as a token, with modified segment embeddings, or dedicated lemma embeddings) significantly improves performance, especially in the English setting. This suggests that directly signaling the target word is crucial.
  • NMT is a viable solution for low-resource languages: Translating existing high-resource language data (English) into a low-resource language (Chinese) using NMT provides a practical way to create training data and achieve competitive results.
  • Challenges in translated data: While NMT is effective, the paper notes that the translated lemma token might not always be the most appropriate translation, impacting performance gains in the Chinese setting compared to English. This highlights the inherent difficulty of fine-grained translation without context.

Key Results

The GX system achieved the following accuracies:

  • English Multilingual Setting: 82.7%
  • Chinese Multilingual Setting: 76.7%

The paper also shows the impact of their proposed modules:

  • In English, adding Lemma Token, Segment $E_c$, and Lemma Embeddings sequentially improved accuracy from 81.6% to 82.7%.
  • In Chinese, the standard fine-tuning achieved the best performance (76.7%), with the introduction of Lemma Token and Lemma Embeddings showing a performance decrease, likely due to translation ambiguities. Segment $E_c$ offered a slight improvement.

Limitations and Future Work

  • Translation Ambiguity: The primary limitation in the Chinese setting is the potential for inaccurate lemma translation due to the lack of context in NMT. This led to the final Chinese system reverting to a simpler fine-tuning approach without lemma-specific enhancements.
  • Data Size Importance: The error analysis suggests that larger datasets for specific parts of speech lead to better performance, underscoring the need for more diverse and extensive training data.
  • Dropout Impact: The study on dropout rates indicates that optimal performance is achieved around a 0.2 dropout rate, with higher rates degrading performance. The paper also notes that dropout has a more stabilizing effect on real (English) corpus compared to pseudo (Chinese) corpus.

Future work could explore more sophisticated NMT techniques that better handle lemma translation or investigate unsupervised methods for generating cross-lingual word sense representations.

Summary updated

BibTeX
@inproceedings{xie2021gx,
  author = {Wanying Xie},
  title = {{GX at SemEval-2021 Task 2: BERT with Lemma Information for MCL-WiC Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure