scaLAR SemEval-2024 Task 1: Semantic Textual Relatednes for English

Anand Kumar, Hemanth KumarNAACL 2024

TL;DR

This paper tackles the challenge of measuring semantic textual relatedness (STR) in English using BERT-based sentence embeddings. The authors explore both supervised fine-tuning and unsupervised approaches, achieving a Spearman correlation of 0.803 with fine-tuning and 0.693 without, demonstrating the effectiveness of pre-trained models for this task.

What Problem Does This Paper Address?

Understanding how closely two pieces of text relate in meaning, known as Semantic Textual Relatedness (STR), is fundamental for many Natural Language Processing (NLP) tasks. While semantic similarity (e.g., paraphrasing) has been extensively studied, STR encompasses a broader range of relationships, including topical relevance, viewpoint alignment, and logical flow. Accurately quantifying this relatedness is crucial for evaluating Large Language Models (LLMs) and improving tasks like question answering and summarization. The paper addresses the need for robust methods to assess STR, particularly in English, and explores the utility of advanced pre-trained language models for this purpose.

Methods

The core of this work revolves around leveraging pre-trained transformer models, specifically BERT, for STR. The authors utilize the bert-base-nli-mean-tokens model from the Sentence-Transformers library. This model is designed to generate fixed-size sentence embeddings by averaging token embeddings, making it suitable for tasks like Semantic Textual Similarity (STS).

Two primary approaches were investigated:

  • Unsupervised Approach (Track B): Here, the pre-trained bert-base-nli-mean-tokens model is used directly to generate sentence embeddings. Cosine similarity is then computed between these embeddings to quantify the relatedness between sentence pairs.
  • Supervised Approach (Track A): This approach involves fine-tuning the bert-base-nli-mean-tokens model on the provided training data. The goal is to adapt the model's embeddings to better capture the specific nuances of semantic relatedness in the target dataset. The fine-tuning process uses a CosineSimilarityLoss with an Adam optimizer.

Datasets

The study utilizes the dataset from the SemEval-2024 STR task. This dataset consists of:

  • Training instances: 5,500 samples, each with a PairID, Text (containing two sentences separated by '/n'), and a Score.
  • Test instances: 250 samples, each with a PairID and Text.

The paper does not introduce a new dataset but rather uses an existing benchmark for evaluating STR.

Key Insights and Ideas

  • Pre-trained models are powerful for STR: The study highlights that even without fine-tuning, BERT-based sentence embeddings combined with cosine similarity can achieve a respectable level of performance in quantifying semantic relatedness.
  • Fine-tuning offers a significant boost: Fine-tuning the BERT model on the specific STR task data leads to a notable improvement in performance, indicating that adapting these models to the target domain is beneficial.
  • STR is more than just similarity: The paper implicitly reinforces the distinction between semantic similarity and the broader concept of semantic relatedness, suggesting that models capable of capturing diverse textual connections are essential.

Key Results

The performance of the models was evaluated using the Spearman rank correlation coefficient, the official metric for the SemEval-2024 STR task.

  • Supervised Approach (Track A): Achieved a Spearman correlation of 0.803.
  • Unsupervised Approach (Track B): Achieved a Spearman correlation of 0.693 using cosine similarity.

The paper notes that the baseline score for the supervised approach was 0.830, and while their proposed approach achieved 0.803, further improvements are possible.

Limitations and Future Work

  • Limitations:
    • The paper mentions that the bert-base-nli-mean-tokens model, while used, is noted as "no longer in use because of its poorer sentence embeddings" in the context of comparing results. This suggests that while it was used for the experiments, it might not be the state-of-the-art choice for sentence embeddings.
  • The supervised approach's score (0.803) was slightly lower than the reported baseline (0.830), indicating room for improvement.
  • Future Work:
    • Alternate approaches: The authors suggest exploring other pre-trained models and potentially increasing the training data size by incorporating more datasets.
  • Zero-shot capabilities: Investigating the zero-shot capabilities of models like GPT could be a promising direction.
  • Model improvements: Further refinement of the supervised approach is suggested to potentially surpass the baseline performance.

Summary updated

BibTeX
@inproceedings{kumar2024scalar,
  author = {Anand Kumar and Hemanth Kumar},
  title = {{scaLAR SemEval-2024 Task 1: Semantic Textual Relatednes for English}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure