What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Study

Mohamed Abdalla, Krishnapriya Vishnubhotla, Saif M. Mohammad

Abstract

The degree of semantic relatedness of two units of language has long been considered fundamental to understanding meaning. Additionally, automatically determining relatedness has many applications such as question answering and summarization. However, prior NLP work has largely focused on semantic similarity, a subset of relatedness, because of a lack of relatedness datasets. In this paper, we introduce a dataset for Semantic Textual Relatedness, STR-2022, that has 5,500 English sentence pairs manually annotated using a comparative annotation framework, resulting in fine-grained scores. We show that human intuition regarding relatedness of sentence pairs is highly reliable, with a repeat annotation correlation of 0.84. We use the dataset to explore questions on what makes sentences semantically related. We also show the utility of STR-2022 for evaluating automatic methods of sentence representation and for various downstream NLP tasks.

Introduction

In Natural Language Processing (NLP), "semantic similarity" and "semantic relatedness" are often used interchangeably, yet they represent distinct concepts. Similarity is a narrow subset of relatedness, focusing on synonyms, hyponyms (e.g., dog and animal), or paraphrases. In contrast, relatedness is a broader category encompassing any functional or topical connection—for instance, the relationship between surgeon and scalpel, or two sentences sharing a common theme despite having different structures.

While datasets for Semantic Textual Similarity (STS) are abundant, sentence-level relatedness datasets have been scarce. Furthermore, existing resources often rely on coarse Likert scales (e.g., 1–5), which are prone to annotator inconsistency. To address this, we introduce STR-2022, a large-scale dataset specifically designed to measure semantic textual relatedness between sentence pairs.

Methodology: Comparative Annotation

To ensure high-quality, fine-grained scores, we employed Best-Worst Scaling (BWS). Rather than assigning an absolute score to a single pair, annotators were presented with four sentence pairs and asked to identify the most and least related pairs. This comparative approach is mathematically more reliable than absolute rating scales, as it mitigates individual biases in numerical interpretation and forces clearer distinctions. These judgments were converted into real-valued scores ranging from 0 (unrelated) to 1 (highly related).

To analyze the drivers of these scores, we utilized the Dice Coefficient for lexical overlap and spaCy for Part-of-Speech (POS) tagging and dependency parsing. This allowed us to investigate whether relatedness is primarily driven by specific word types (e.g., nouns vs. verbs) or syntactic roles (e.g., subjects vs. objects).

The STR-2022 Dataset

The STR-2022 dataset comprises 5,500 English sentence pairs sampled from seven diverse sources to ensure a wide range of structures and formality levels.

Composition of the STR-2022 dataset across various sources and pair types.

The sources include book reviews (Goodreads), machine-translated paraphrases (ParaNMT), image captions (SNLI), and tweets (Stance), among others. This diversity ensures the dataset captures various facets of how humans perceive relatedness across different domains.

Analysis and Key Insights

Our empirical study yielded several significant findings:

  • Human Intuition is Robust: Despite the subjective nature of "relatedness," human judgments are remarkably consistent. The dataset achieved a split-half reliability (SHR) of 0.84, confirming that relatedness is a measurable linguistic phenomenon.
  • Nouns as Anchors: Relatedness scores correlate more strongly with the presence of related nouns (particularly proper nouns) and objects than with verbs or subjects. Humans tend to rate sentences as highly related if they share a common topic, even if the actions described differ significantly.
  • The Similarity Gap: The correlation between STR-2022 scores and traditional similarity scores was notably low in certain ranges. This suggests that models optimized solely for similarity fail to capture a significant portion of human textual intuition.

Model Evaluation and Results

We evaluated several baseline and state-of-the-art models on the STR-2022 dataset. Simple lexical overlap (Dice) provides a baseline correlation of 0.57.

Performance comparison of baseline, unsupervised, and supervised models on the STR-2022 dataset.

As shown in the results, standard unsupervised embeddings (e.g., vanilla BERT or Word2Vec) perform modestly, with the best unsupervised model (Word2Vec mean) reaching a Spearman correlation of 0.60. However, fine-tuning models like RoBERTa on STR-2022 significantly boosts performance to 0.83, nearly matching human reliability. This underscores the necessity of dedicated relatedness data for training effective NLP models.

Conclusion and Limitations

STR-2022 provides a rigorous foundation for studying semantic relatedness. While the dataset demonstrates high reliability, we acknowledge limitations regarding its focus on US-based English annotators and the need for cross-linguistic studies. Future research will explore how logical reasoning and narrative flow contribute to relatedness and how these insights can be integrated into more sophisticated sentence representations.

Summary updated

Abstract

The degree of semantic relatedness of two units of language has long been considered fundamental to understanding meaning. Additionally, automatically determining relatedness has many applications such as question answering and summarization. However, prior NLP work has largely focused on semantic similarity, a subset of relatedness, because of a lack of relatedness datasets. In this paper, we introduce a dataset for Semantic Textual Relatedness, STR-2022, that has 5,500 English sentence pairs manually annotated using a comparative annotation framework, resulting in fine-grained scores. We show that human intuition regarding relatedness of sentence pairs is highly reliable, with a repeat annotation correlation of 0.84. We use the dataset to explore questions on what makes sentences semantically related. We also show the utility of STR-2022 for evaluating automatic methods of sentence representation and for various downstream NLP tasks. Our dataset, data statement, and annotation questionnaire can be found at: https://doi.org/10.5281/zenodo.7599667
BibTeX
@inproceedings{abdalla2023what,
  author = {Mohamed Abdalla and Krishnapriya Vishnubhotla and Saif M. Mohammad},
  title = {{What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Study}},
  booktitle = {EACL},
  year = {2023},
  url = {https://openreview.net/forum?id=gUGGV4kNub},
}
Figure