TL;DR
This paper explores how to best predict semantic relatedness between English sentences by combining traditional text features like TF-IDF and syntactic information with various distance-based measures. The authors found that a Gradient Boost model, leveraging these combined features, achieved a strong Spearman score of 0.8371 on the SemEval-2024 Task 1 dataset, demonstrating the effectiveness of this hybrid approach.
What Problem Does This Paper Address?
Accurately understanding the meaning and relationship between sentences is a fundamental challenge in Natural Language Processing (NLP). This task, known as Semantic Textual Relatedness (STR), has broad applications, from improving search engines and question-answering systems to enhancing text summarization. While word embeddings and deep learning models have made strides, capturing the nuanced semantic connections between sentences remains difficult. This paper aims to identify which features are most crucial for predicting STR and to build a system that effectively leverages them, going beyond simple word similarity.
Methods
The researchers employed a multi-faceted approach to feature extraction and model training. They deliberately avoided complex preprocessing steps to preserve sentence structure, which they argue is vital for information retrieval and semantic identification. Notably, they also excluded Large Language Models (LLMs) from their primary experiments, citing challenges in interpreting their internal workings and feature contributions.
Their feature engineering involved:
- TF-IDF: Standard Term-Frequency Inverse Document Frequency was used to represent word importance.
- Syntactic Features: They extracted features based on words sharing the same dependency roles within sentence pairs, looking at probabilities of exact word matches, unique words, and related words.
- Distance-Based Measures: A variety of similarity and distance metrics were applied, including Jaccard distance, Cosine similarity, Levenshtein distance, and Word Mover's Distance (WMD). These measures capture relationships based on shared vocabulary, term vectors, and the "travel" distance of words between documents.
- Sentence Embeddings: Hidden vectors from a pre-trained Sentence-BERT model were used to compute cosine similarity.
These features were then fed into traditional machine learning models, with a focus on Gradient Boost due to its ability to handle non-linear relationships and imbalanced datasets.
Datasets
The study utilized the SemEval-2024 Task 1 dataset, which consists of sentence pairs annotated with semantic relatedness scores ranging from 0 (unrelated) to 1 (related). The dataset was divided into training, development, and testing sets. The paper provides statistics on the distribution of scores, noting a significant number of instances with scores of 1.0 and 0.0, as well as those falling into the >0.80 and <0.50 ranges.
Key Insights and Ideas
- Feature Combination is Key: The most significant insight is that combining diverse features – syntactic, TF-IDF, and distance-based measures – yields substantially better results than relying on any single category.
- Traditional Models Still Relevant: Despite the rise of LLMs, traditional models like Gradient Boost, when equipped with well-engineered features, can achieve competitive performance on STR tasks.
- Syntactic Features Add Value: While not the sole driver of performance, syntactic features, particularly those related to word dependency roles, contributed to the overall predictive power of the models.
- Preserving Sentence Structure Matters: The decision to avoid extensive preprocessing highlights the importance of maintaining the original sentence structure for capturing semantic nuances.
Key Results
The paper reports the following key performance metrics, primarily using the Spearman correlation coefficient:
- Training Phase: The Gradient Boost model achieved a Spearman score of 0.8527 during the training phase.
- Development Phase: The Gradient Boost model maintained a score of 0.8527 on the development set.
- Test Phase: The final Gradient Boost model achieved a Spearman score of 0.8371, ranking 10th among participating teams. This score was notably higher than the provided baseline (0.8300).
- Individual Feature Performance: When using only syntactic features, the maximum Spearman score was 0.32. Using only TF-IDF features resulted in a score of 0.533. Combining SBERT embeddings with syntactic features yielded a score of 0.70. The full combination of strategies achieved the highest scores.
Limitations and Future Work
The authors acknowledge several limitations:
- Absence of Word Sense Disambiguation (WSD): The lack of WSD algorithms might have limited the models' ability to fully grasp nuanced meanings.
- Exclusion of Transformer Models: While Transformers might offer superior performance, their "black box" nature made them less suitable for this study, which aimed for explainability.
- Dataset Merging: The decision not to merge training and development sets for the final training phase might have led to missed opportunities for improvement.
Future work could involve:
- Incorporating WSD algorithms.
- Exploring Transformer models for their potential performance gains, perhaps with methods for improving explainability.
- Merging datasets or augmenting them to potentially boost predictive capabilities.
- Developing approaches to better distinguish between Semantic Textual Relatedness (STR) and Semantic Textual Similarity (STS).
Summary updated