Introduction
Semantic Textual Similarity (STS) is a core challenge in Natural Language Processing (NLP) that involves determining the degree of semantic equivalence between two sentences. Unlike binary paraphrase detection, STS requires a nuanced score, typically on a scale from 0 (no similarity) to 5 (semantic equivalence). This task is foundational for high-level applications such as Question Answering (QA), Machine Translation (MT) evaluation, and document summarization.
The DLS@CU team developed a supervised system for the SemEval-2016 STS task that leverages a "less is more" approach. By combining a sophisticated monolingual word aligner with neural sentence embeddings into a simple ridge regression model, they achieved state-of-the-art results, demonstrating that a small set of high-quality features can outperform more complex feature sets.
Methodology
The system architecture is built on Ridge Regression, a linear regression model with L2 regularization to prevent overfitting. The authors submitted three variations (runs) of their system, focusing on four primary feature types:
- Word Alignment Proportion: Utilizing a custom monolingual aligner, this feature pairs words across sentences based on exact matches, contextual similarity, and entries in the Paraphrase Database (PPDB). The proportion of aligned "content words" (e.g., nouns, verbs) serves as a strong indicator of similarity.
- Sentence Embeddings: To capture semantic relationships that word alignment might miss (e.g., "feline" vs. "cat"), the system uses 400-dimensional neural word vectors. Each sentence is represented as the sum of its word vectors, and the cosine similarity between the two resulting sentence vectors is calculated.
- N-gram Overlap: This feature measures shared sequences of words (1-3 grams) or characters (3-4 grams) to capture local similarities and account for spelling errors.
- Soft Cardinality: A set-theory-based measure that calculates similarity by considering the unique information each word contributes to a set, rather than simple word counts.
Datasets and Evaluation
The models were trained on over 14,000 human-annotated sentence pairs from previous SemEval competitions (2012–2015). Evaluation for the 2016 task involved 1,186 new pairs across five distinct domains: Answer-Answer (Q&A forums), Headlines (news), Plagiarism (plagiarized versions), Postediting (MT vs. human-edited), and Question-Question (forum variations).
Results and Discussion
The system's most successful configuration, Run 1, utilized only word alignment and sentence embedding features. This model achieved a 73.6% weighted mean Pearson correlation with human annotations. Interestingly, adding more complex features like n-gram overlaps and soft cardinality (Runs 2 and 3) slightly decreased performance, reinforcing the efficacy of the core feature set.
Feature Importance and Ablation
An ablation study revealed that while both alignment and embeddings are strong individual predictors, their combination provides a significant performance boost.
As shown in the table above, the Alignment feature alone achieved a 70.8% correlation, while Embeddings achieved 69.9%. The combined Run 1 outperformed both, suggesting that these features capture complementary semantic information.
Performance Across Domains
The system demonstrated high sensitivity to the text domain. It performed exceptionally well on news headlines and machine translation data (80-84% correlation) but struggled with informal, conversational text from Q&A forums (55-65% correlation).
Conclusion and Future Work
The DLS@CU system proves that a streamlined supervised model can achieve competitive results in STS. However, several challenges remain. The current model struggles with units larger than single words (e.g., idioms like "kick the bucket"), lacks nuance in handling stop words, and cannot yet account for fine-grained semantics like negation or modality. Future work will focus on domain adaptation—training vectors specifically for technical or medical forums—and developing methods to handle phrasal semantics and logical polarity.
Summary updated