Lotus at SemEval-2021 Task 2: Combination of BERT and Paraphrasing for English Word Sense Disambiguation

Niloofar Ranjbar, Hossein ZeinaliACL 2021

TL;DR

This paper tackles the challenge of Word Sense Disambiguation (WSD) by leveraging BERT's contextual understanding. The authors explore augmenting BERT with external knowledge from WordNet and by generating paraphrases of sentences to improve its ability to distinguish between different meanings of a word in context.

What Problem Does This Paper Address?

Understanding the meaning of a word in context is crucial for many Natural Language Processing (NLP) tasks. However, a single word can have multiple meanings, and distinguishing between them based solely on surrounding text can be difficult, especially when sentences share a similar subject but use the target word differently. This is the core problem addressed by the Word-in-Context (WiC) task, specifically in the SemEval-2021 Task 2. The challenge is to determine if a target word in two different sentences is used with the same meaning or not. This paper focuses on the English portion of this multilingual task.

Methods

The authors propose four main methods, all building upon the pre-trained BERT model:

  • BERT Method (3.1): This is the baseline approach. The two sentences are concatenated with special tokens ([TGT] around the target word and [SEP] between sentences) and fed into BERT. The output of BERT is then used to classify whether the word sense is the same or different.
  • BERT + WordNet Method (3.2): This method augments the BERT input with information from WordNet. For each synset (meaning) of the target word, a score is calculated based on its relevance to the context using a path distance metric. These relevance scores, along with Lexnames (WordNet's supersenses), are added as extra input to BERT.
  • BERT + Paraphrase Sentences Method (3.3.1): Here, the authors generate paraphrases of the original sentences using machine translation (English -> Spanish -> English). These paraphrased sentences are then used as additional input to BERT, aiming to expose the model to variations in phrasing that might highlight different meanings.
  • BERT + Paraphrase Words Method (3.3.2): This is a refinement of the paraphrase method. Instead of using the entire paraphrased sentence, this approach identifies words that have changed in the paraphrase and attempts to find their original counterparts in the main sentence. These identified changed words are then concatenated with the original sentence.

Datasets

The experiments were conducted using the MCL-WiC dataset for English:

  • Training Set: 8000 sentence pairs.
  • Development Set: 1000 sentence pairs (used for hyperparameter tuning, specifically finding the best number of fine-tuning iterations).
  • Test Set: 1000 sentence pairs (used for final evaluation).

Key Insights and Ideas

  • The Power of Context Augmentation: The core idea is that BERT, while powerful, can benefit from explicit external information or varied phrasings to disambiguate word senses.
  • Paraphrasing as a Meaning Revealer: Generating paraphrases, especially at the sentence level, proved to be an effective way to expose BERT to different contextual nuances of a word, leading to improved performance. The intuition is that paraphrasing can alter sentence structure and wording while preserving meaning, or subtly shift meaning, which BERT can then learn to differentiate.
  • WordNet's Nuanced Contribution: While WordNet provides valuable lexical information, simply adding raw synset information or Lexnames didn't significantly boost performance over the baseline BERT. This suggests that the way this external knowledge is integrated and represented is critical.
  • Challenges in Word-Level Paraphrasing: The method focusing on paraphrased words showed only marginal improvement, highlighting the difficulty in accurately identifying original word meanings after paraphrasing and the potential for introduced discontinuities.

Key Results

The paper reports accuracy percentages on the MCL-WiC test set, averaged over 5 runs:

  • BERT Method (3.1): Achieved an average accuracy of 84.62%.
  • BERT + WordNet Method (3.2): Achieved an average accuracy of 84.78%, showing a marginal improvement over the baseline.
  • BERT + Paraphrase Sentences Method (3.3.1): Achieved the highest average accuracy of 85.78%, demonstrating the effectiveness of sentence-level paraphrasing.
  • BERT + Paraphrase Words Method (3.3.2): Achieved an average accuracy of 84.76%, which was only slightly better than the baseline and WordNet methods.

The paraphrase sentence method also exhibited a lower standard deviation (0.571) compared to the baseline (0.688) and paraphrase words (1.378), suggesting greater robustness.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • WordNet Integration: The method of incorporating WordNet information might not be optimal. Future work could explore adding only the first few synsets, or using lemmas and glosses instead of Lexnames.
  • Paraphrase Generation: The effectiveness of the paraphrase method is highly dependent on the quality of the generated paraphrases. The authors suggest exploring different machine translation models or paraphrasing techniques.
  • Word-Level Paraphrase Challenges: The difficulties encountered in the "Paraphrase Words" method suggest that more sophisticated techniques are needed to accurately map changed words back to their original meanings.
  • Time Constraints: The authors mention that due to time limitations, they couldn't explore all avenues thoroughly, leaving room for further investigation.

Summary updated

BibTeX
@inproceedings{ranjbar2021lotus,
  author = {Niloofar Ranjbar and Hossein Zeinali},
  title = {{Lotus at SemEval-2021 Task 2: Combination of BERT and Paraphrasing for English Word Sense Disambiguation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure