Zhestyatsky at SemEval-2021 Task 2: ReLU over Cosine Similarity for BERT Fine-tuning

Boris Zhestiankin, Maria PonomarevaACL 2021

TL;DR

This paper tackles the challenge of disambiguating word meanings in context, a crucial step for deeper language understanding. The authors found that fine-tuning pre-trained language models like BERT and XLM-RoBERTa using a simple Cosine Similarity approach, activated by ReLU, significantly outperforms more complex methods, achieving the fourth-best score on the multilingual WiC task.

What Problem Does This Paper Address?

Understanding the meaning of a word often depends heavily on its surrounding context. While early word embeddings captured word meanings as static vectors, they struggled with polysemy (words with multiple meanings). More recent contextual embeddings, like those from BERT, have revolutionized NLP by generating word representations that adapt to their context. However, directly evaluating how well these embeddings capture semantic nuances remains a challenge. The SemEval-2021 Task 2 (MCL-WiC) was designed to address this by providing a framework to evaluate word meaning disambiguation across multiple languages. This paper focuses on the English sub-track of this task, aiming to build a system that can accurately determine if a target word has the same meaning in two different sentences.

Methods

The core of this work lies in fine-tuning pre-trained language models (BERT and XLM-RoBERTa) for the Word-in-Context (WiC) task. The authors explored two main architectural approaches:

  1. Multilayer Perceptron (MLP) Architecture: This approach involved concatenating the embeddings of the target word from both sentences and feeding them into an MLP. They also experimented with including the [CLS] token's embedding, which represents the entire sentence, to enrich the input. However, this method proved prone to overfitting, especially with additional linear layers.
  2. Cosine Similarity Architecture: This proved to be the winning strategy. Instead of a complex network, this method directly calculates the Cosine Similarity between the target word embeddings from the two sentences. This similarity score is then passed through a ReLU activation function, and the output is used to predict whether the words have the same meaning. Crucially, this approach does not introduce new trainable weights; it only fine-tunes the existing weights of the pre-trained models. They also experimented with a sigmoid activation as an alternative to ReLU.

Datasets

The primary datasets used were the English training and development data provided for the SemEval-2021 Task 2 (MCL-WiC) EN-EN sub-track. To further enhance performance, they also incorporated the WiC dataset, which is part of the SuperGLUE benchmark.

Key Insights and Ideas

  • Simplicity Wins: The most surprising finding is that a straightforward Cosine Similarity approach, when combined with ReLU activation, significantly outperforms more complex MLP architectures for this task. This suggests that the rich contextual information already present in pre-trained models is sufficient, and complex downstream layers can lead to overfitting.
  • ReLU's Role: The ReLU activation function plays a critical role in making the Cosine Similarity approach effective. It helps in learning the similarity from limited data without causing instant overfitting.
  • Pre-trained Power: The inherent ability of pre-trained models like BERT and XLM-RoBERTa to associate similar words with close vectors is a strong foundation. Fine-tuning these models, rather than building entirely new ones, is highly effective.
  • Dataset Representativeness: The authors observed that the validation dataset was more challenging than the test dataset, indicating its representativeness.
  • Data Augmentation is Key: Using additional data (like the WiC dataset) beyond the competition organizers' provided data led to a notable performance improvement.

Key Results

The authors' best model, utilizing BERT-large-cased embeddings with the Cosine Similarity architecture and ReLU activation, achieved an accuracy of 92.7% on the EN-EN sub-track of SemEval-2021 Task 2. This placed them fourth on the leaderboard, just 0.6% behind the winner.

Limitations and Future Work

  • Dataset Peculiarity: The authors noted a peculiarity in the dataset where repeating first sentences in pairs could lead to consistent mispredictions. If the model made an error on one instance of a lemma, it was likely to err on the second instance with the same first sentence. This makes it difficult to definitively attribute errors to specific lemmas or contexts.
  • Ensemble Potential: The error analysis revealed that most mislabeled instances were correctly predicted by at least one of the other models tested. This suggests that an ensemble of their models could yield even better results.
  • Future Directions: While not explicitly detailed, the success of the Cosine Similarity approach with ReLU activation opens avenues for further exploration in other context-sensitive NLP tasks. The paper also hints at the potential of ensemble methods.

Summary updated

BibTeX
@inproceedings{zhestiankin2021zhestyatsky,
  author = {Boris Zhestiankin and Maria Ponomareva},
  title = {{Zhestyatsky at SemEval-2021 Task 2: ReLU over Cosine Similarity for BERT Fine-tuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure