TL;DR
This paper tackles the challenge of measuring semantic textual similarity (STS) by leveraging machine translation (MT) evaluation metrics as features for regression models. The authors demonstrate that by combining various MT metrics with boosted tree ensembles, they can achieve competitive results, outperforming the median performance of participating systems in the SemEval-2016 STS task.
What Problem Does This Paper Address?
The core problem this paper addresses is accurately determining how similar two pieces of text are in meaning, a task known as Semantic Textual Similarity (STS). While humans excel at this, building automated systems is complex. Traditional STS systems often rely on lexical overlap or complex linguistic features. This paper explores a novel approach by repurposing metrics commonly used to evaluate the quality of machine translation (like BLEU, METEOR, etc.) as features for STS. The hypothesis is that these MT metrics, designed to assess how well a translation matches a reference, can also capture semantic nuances between two independent text snippets. The challenge lies in effectively integrating these diverse MT metrics into a robust STS model.
Methods
The SAARSHEFF system employs a feature-based regression approach. They extract a rich set of features from text pairs using various Machine Translation evaluation metrics. These metrics fall into three main categories:
- Lexical/Syntactic Features (Asiya toolkit): These include n-gram overlaps (character and token level, up to 5-grams) using cosine similarity and Jaccard Index, as well as shallow parsing and Named Entity (NE) annotation overlaps.
- Syntactic/Semantic Features (BEER): This metric considers character n-gram overlaps, Kendall tau distance for word order, and abstract ordering patterns from tree factorization, differentiating between function and non-function words.
- Semantic Features (METEOR): METEOR aligns translations to references and uses unigram matching based on surface forms, stems, synonyms, and paraphrases.
- Deep Neural Network Features (ReVal): This metric uses cosine similarity between dense vector representations of sentences generated by a Tree-based LSTM.
These features are then used to train three regression models: a linear regressor, a boosted tree regressor, and an eXtreme Gradient Boosted tree regressor (XGBoost). The boosted tree models, particularly XGBoost, are highlighted for their ability to capture complex interactions between features.
Datasets
The authors annotated existing STS datasets from SemEval-2012 to 2015 with the described MT evaluation metrics. These annotated datasets were then used for training their models. The evaluation was performed on the SemEval-2016 English STS task datasets.
Key Insights and Ideas
- MT Metrics as Powerful STS Features: The paper strongly suggests that MT evaluation metrics, despite being designed for a different task, are highly effective features for STS. This opens up a new avenue for STS research by leveraging existing, well-established MT evaluation tools.
- Boosted Trees Excel: The use of boosted tree ensembles, especially XGBoost, proved superior to a simple linear regressor. This indicates that the relationships between MT metrics and semantic similarity are non-linear and benefit from more complex modeling.
- Challenges at Extremes: The study highlights that MT metrics struggle with texts that are either completely unrelated (score 0) or semantically equivalent but lexically very different (score 5). This is because MT metrics often penalize lexical divergence, even when the meaning is preserved or when the lack of similarity is clear.
- Semantic Adequacy is Key: The authors emphasize that for STS, semantic adequacy (capturing the meaning) is paramount, and this aspect is often overlooked in standard MT evaluation which can heavily rely on surface-level similarity.
Key Results
The SAARSHEFF submission achieved competitive results in the SemEval-2016 English STS task. Their baseline linear model outperformed the median score for most domains, except for the "answer-answer" domain. The boosted tree models performed even better, with the XGBoost model achieving the highest Pearson correlation scores among their submissions. Specifically, their best model achieved a Pearson correlation of 0.77807 for the "All" domain, which was better than the median score of participating teams. The paper also includes a detailed L1 error analysis showing their XGBoost model's performance across different gold standard similarity scores, illustrating its strengths and weaknesses.
Limitations and Future Work
The paper acknowledges that their models perform poorly on the extreme ends of the similarity scale (scores 0 and 5). This is attributed to the inherent limitations of MT metrics, which tend to penalize lexical differences even when meanings are disparate or identical. The authors suggest that future work could explore more sophisticated ways to handle these extreme cases, potentially by developing specialized features or models that are more sensitive to absolute semantic dissimilarity or equivalence, rather than relying solely on comparative MT metrics. They also mention releasing their MT metrics annotations and system implementations, which could facilitate further research in this area.
Summary updated