Linguistically Conditioned Semantic Textual Similarity

Jingxuan Tu, Keer Xu, Liulu Yue, Bingyang Ye, Kyeongmin Rim, James PustejovskyACL 2024 · DOI

Abstract

Semantic Textual Similarity (STS) is a fundamental task in Natural Language Processing (NLP), traditionally measuring the degree of semantic equivalence between two sentences. However, meaning is often context-dependent. Conditional STS (C-STS) was introduced to address this by evaluating similarity relative to a specific aspect or condition. Despite its potential, recent research from Brandeis University reveals significant structural and annotative flaws in the standard C-STS benchmark, finding that 55% of validation labels were problematic. This paper proposes a novel framework that treats conditions as Question-Answering (QA) tasks and introduces Typed-Feature Structures (TFS) to provide a more objective linguistic foundation for similarity measurement.

Introduction: The Challenges of Conditional STS

While the concept of C-STS—asking "How similar are these sentences given a specific aspect?"—is promising, the existing benchmark dataset suffers from three primary issues:

  1. Incommensurable Mapping: Binary conditions (e.g., "is there a dog?") are forced onto a 5-point similarity scale without clear logic.
  2. Ambiguity: Single-word conditions like "the table" fail to specify which attributes (size, material, position) should be compared.
  3. Annotation Errors: A lack of rigorous guidelines led to high levels of noise and subjectivity in the original labels.
Figure 1: Ambiguity and labeling challenges in the original C-STS dataset, illustrating how vague conditions lead to inconsistent similarity scores.

Methodology

The researchers addressed these flaws through a multi-pronged approach designed to increase objectivity and reduce reasoning complexity.

Expert Re-annotation and Error Detection

Four trained linguists re-labeled the C-STS validation set (2,620 samples), establishing strict rules for binary conditions and inference degrees. Additionally, an automated pipeline using K-means clustering and embedding-based ranking was developed to flag likely errors in the original dataset, achieving an 82.4 F1 score in error identification.

QA-Facilitated C-STS

The core innovation involves transforming the fuzzy "condition" into a structured query. By using a Large Language Model (LLM) to answer a specific question for each sentence (e.g., "What is the number of people in Sentence A?"), the task becomes a comparison of two concise answers. This allows the model to ignore irrelevant "noise" in the original sentences.

Typed-Feature Structures (TFS)

To move beyond subjective "vibes," the authors proposed representing entities as Typed-Feature Structures. By decomposing an entity (e.g., "Animal") into specific attributes like type, color, and count, similarity can be calculated through a weighted mathematical approach, providing a more nuanced 5-point score.

Experimental Results

The impact of the relabeled dataset and the QA-based approach is significant across all model architectures. When evaluated on the cleaned "gold standard" validation set, performance metrics saw a dramatic increase.

Figure 5: Spearman's Correlation comparison showing the performance leap across various models when moving from the original noisy labels to the relabeled dataset.

Using the QA-based method, even smaller models achieved massive gains. For instance, on a tri-encoder setup, performance jumped from a Spearman correlation of 12.8 to 73.4. Furthermore, the QA-based approach allowed efficient models like Flan-T5 to achieve results comparable to much larger proprietary models like GPT-4.

Conclusion

The discovery of a 55% discrepancy rate in a major benchmark serves as a critical reminder of the importance of dataset integrity. By reframing conditions as queries and utilizing structured linguistic representations like TFS, the researchers have provided a path toward more robust and interpretable semantic evaluation. Future work will focus on scaling these structured representations to larger training sets and mitigating occasional LLM hallucinations in the QA generation phase.

Summary updated

BibTeX
@inproceedings{tu2024linguistically,
  author = {Jingxuan Tu and Keer Xu and Liulu Yue and Bingyang Ye and Kyeongmin Rim and James Pustejovsky},
  title = {{Linguistically Conditioned Semantic Textual Similarity}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.64},
}
Figure