NLP_STR_teamS at SemEval-2024 Task1: Semantic Textual Relatedness based on MASK Prediction and BERT Model

Lianshuang Su, Xiaobing ZhouNAACL 2024

TL;DR

This paper presents a BERT-based approach for Semantic Textual Relatedness (STR) on African and Asian languages, framed as a masked language modeling task. By augmenting data with synonym replacement and using a prompt template, the team achieved a Spearman correlation of 0.809 on the SemEval-2024 Task 1 English track.

What Problem Does This Paper Address?

Semantic Textual Relatedness (STR) is crucial for understanding the nuanced relationships between sentences, going beyond simple similarity to encompass themes, intentions, and emotions. While NLP has made strides in semantic similarity, datasets and robust methods for broader relatedness, especially across diverse languages, have been less explored. This paper tackles the challenge of accurately measuring STR, particularly for African and Asian languages, by leveraging powerful pre-trained language models like BERT and addressing data scarcity through augmentation.

Methods

The core of the approach involves using the BERT pre-training model for encoding and feature extraction of sentence pairs. To adapt BERT for STR, the authors employ a "prompt tuning" technique. They construct a specific prompt template: "The correlation of the next two sentences sent0 and sent1 is [MASK]." This transforms the STR task into a masked language modeling (MLM) problem, where BERT predicts the missing correlation score. For data augmentation, they utilize a synonym replacement method to increase the size of the training dataset, aiming to improve generalization and robustness. The model is trained using a mean squared error (MSE) loss function, suitable for regression tasks.

Datasets

The primary dataset used is from the SemEval-2024 Task 1, "Semantic Textual Relatedness for African and Asian Languages." This task provides sentence pairs with associated STR scores. The dataset is divided into training, development (dev), and test sets. The authors specifically focus on the English track (Track A). They report that the initial training set size was 5500 instances, which was then augmented to 11000 instances using their synonym replacement method.

Key Insights and Ideas

  • Framing STR as MLM: The novel idea of framing STR as a masked language modeling task using a carefully crafted prompt template is a key contribution. This allows leveraging BERT's pre-trained capabilities for understanding context and predicting missing information.
  • Prompt Tuning for STR: The use of prompt tuning, specifically the "The correlation of the next two sentences sent0 and sent1 is [MASK]." template, effectively bridges the gap between general language understanding and the specific STR task.
  • Synonym Replacement for Augmentation: While data augmentation is common, the specific choice of synonym replacement for STR is interesting. The authors acknowledge that this method, while increasing data size, can introduce inaccuracies.
  • Dev vs. Test Set Performance Discrepancy: The observation that performance improved on the development set after data augmentation but declined on the test set is a significant insight. This highlights potential overfitting or issues with the augmentation strategy's generalization.

Key Results

The team achieved a Spearman Correlation of 0.809 on the test set for the English track of SemEval-2024 Task 1.

  • Before text data augmentation, the dev set score was 0.819 and the test set score was 0.832.
  • After text data augmentation, the dev set score improved to 0.820, but the test set score dropped to 0.809.

Limitations and Future Work

The authors acknowledge that the synonym replacement method for data augmentation can introduce inaccuracies and lead to biased estimates. They also note that the significant size difference between the training, development, and test sets might have contributed to the performance drop on the test set after augmentation.

Future work suggestions include:

  • Developing methods that are more context-aware and involve manual intervention to address augmentation errors.
  • Expanding the size of the development set and reducing the size discrepancies between all data splits.
  • Exploring more powerful pre-trained language models.

Summary updated

BibTeX
@inproceedings{su2024nlp,
  author = {Lianshuang Su and Xiaobing Zhou},
  title = {{NLP\_STR\_teamS at SemEval-2024 Task1: Semantic Textual Relatedness based on MASK Prediction and BERT Model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure