Abstract
Semantic Textual Similarity (STS) is a fundamental task in Natural Language Processing (NLP), traditionally measuring the degree of semantic equivalence between two sentences. However, meaning is often context-dependent. Conditional STS (C-STS) was introduced to address this by evaluating similarity relative to a specific aspect or condition. Despite its potential, recent research from Brandeis University reveals significant structural and annotative flaws in the standard C-STS benchmark, finding that 55% of validation labels were problematic. This paper proposes a novel framework that treats conditions as Question-Answering (QA) tasks and introduces Typed-Feature Structures (TFS) to provide a more objective linguistic foundation for similarity measurement.
Introduction: The Challenges of Conditional STS
While the concept of C-STS—asking "How similar are these sentences given a specific aspect?"—is promising, the existing benchmark dataset suffers from three primary issues:
- Incommensurable Mapping: Binary conditions (e.g., "is there a dog?") are forced onto a 5-point similarity scale without clear logic.
- Ambiguity: Single-word conditions like "the table" fail to specify which attributes (size, material, position) should be compared.
- Annotation Errors: A lack of rigorous guidelines led to high levels of noise and subjectivity in the original labels.
Methodology
The researchers addressed these flaws through a multi-pronged approach designed to increase objectivity and reduce reasoning complexity.
Expert Re-annotation and Error Detection
Four trained linguists re-labeled the C-STS validation set (2,620 samples), establishing strict rules for binary conditions and inference degrees. Additionally, an automated pipeline using K-means clustering and embedding-based ranking was developed to flag likely errors in the original dataset, achieving an 82.4 F1 score in error identification.
QA-Facilitated C-STS
The core innovation involves transforming the fuzzy "condition" into a structured query. By using a Large Language Model (LLM) to answer a specific question for each sentence (e.g., "What is the number of people in Sentence A?"), the task becomes a comparison of two concise answers. This allows the model to ignore irrelevant "noise" in the original sentences.
Typed-Feature Structures (TFS)
To move beyond subjective "vibes," the authors proposed representing entities as Typed-Feature Structures. By decomposing an entity (e.g., "Animal") into specific attributes like type, color, and count, similarity can be calculated through a weighted mathematical approach, providing a more nuanced 5-point score.
Experimental Results
The impact of the relabeled dataset and the QA-based approach is significant across all model architectures. When evaluated on the cleaned "gold standard" validation set, performance metrics saw a dramatic increase.
Using the QA-based method, even smaller models achieved massive gains. For instance, on a tri-encoder setup, performance jumped from a Spearman correlation of 12.8 to 73.4. Furthermore, the QA-based approach allowed efficient models like Flan-T5 to achieve results comparable to much larger proprietary models like GPT-4.
Conclusion
The discovery of a 55% discrepancy rate in a major benchmark serves as a critical reminder of the importance of dataset integrity. By reframing conditions as queries and utilizing structured linguistic representations like TFS, the researchers have provided a path toward more robust and interpretable semantic evaluation. Future work will focus on scaling these structured representations to larger training sets and mitigating occasional LLM hallucinations in the QA generation phase.
Summary updated