TL;DR
This paper tackles the challenge of accurately predicting semantic textual relatedness (STR) between sentence pairs, a crucial task for many NLP applications. The authors fine-tuned a BERT-based model (STS-BERT) and achieved state-of-the-art results on the SemEval-2024 Task 1 dataset, outperforming traditional machine learning and other deep learning approaches.
What Problem Does This Paper Address?
Predicting how semantically related two pieces of text are is a fundamental problem in Natural Language Processing (NLP). This task, known as Semantic Textual Relatedness (STR), is vital for improving search engines, question-answering systems, text summarization, and machine translation. While semantic similarity has been explored, true relatedness is more nuanced, encompassing broader connections beyond mere synonymy. Existing models often struggle with ambiguous language, cultural context, and the ever-evolving nature of language. The SemEval-2024 Task 1 (Track A3) specifically aims to address this by providing a new dataset for STR prediction.
Methods
The authors explored a range of models, from traditional machine learning (Linear Regression, Random Forest, XGBoost) to deep learning (LSTM, Bi-LSTM) and, crucially, transformer-based models. Their core approach involved fine-tuning a pre-trained BERT model, which they dubbed "STS-BERT." For transformer models like BERT and RoBERTa, they extracted feature vectors from the pooling layer to represent sentence pairs. These embeddings were then fed into a cosine similarity function to predict relatedness scores. They also employed Mean Squared Error (MSE) as a loss function, optimizing parameters using gradient descent. For traditional and deep learning models, they experimented with both Word2Vec and FastText embeddings.
Datasets
The primary dataset used was the SemEval-2024 Task 1 (Track A3) dataset, which contains 5500 training and 250 validation samples. To augment their training data and improve model robustness, they also incorporated three external datasets:
- STSB multilingual benchmark dataset (May, 2021)
- SemEval-2014 Task 1 dataset (Marelli et al., 2014)
- SemEval-2015 Task 2 dataset (Agirre et al., 2015)
A key challenge noted was the differing similarity score ranges across these datasets (0.0-5.0 for external datasets vs. 0.0-1.0 for the competition dataset), which required careful normalization.
Key Insights and Ideas
- Transformer Dominance: The paper strongly reinforces the superiority of transformer-based models (like BERT and RoBERTa) over traditional machine learning and even earlier deep learning architectures for complex semantic tasks.
- Fine-tuning is Key: The success of their STS-BERT model highlights the power of fine-tuning large pre-trained language models on specific downstream tasks like STR.
- Data Normalization Matters: The authors encountered performance degradation due to the need to normalize similarity scores from diverse datasets, underscoring the importance of consistent data preprocessing.
- Beyond Similarity: The focus on "relatedness" rather than just "similarity" is crucial, as it allows for a broader understanding of textual connections.
Key Results
The STS-BERT model achieved the highest performance on the SemEval-2024 Task 1 test dataset, securing a Spearman correlation coefficient of 0.810. This placed them 25th out of all participants. In comparison, the best machine learning model (XGBoost with Word2Vec) achieved 0.3178, and the best deep learning model (BiLSTM with Word2Vec) achieved 0.4990. RoBERTa, another transformer model, scored 0.749.
Limitations and Future Work
The authors acknowledge that the integration of external datasets with varying score ranges led to some precision loss and affected performance. They also identified sentence transformation (e.g., simple to complex form) as a challenge that their model struggled to detect.
For future work, they plan to:
- Implement alternative strategies to address the data normalization issues.
- Expand their work to other tracks of SemEval-2024 Task 1 (Tracks B and C) for more comprehensive findings.
Summary updated