TL;DR
This paper introduces an attention-based convolutional neural network (CNN) for measuring semantic textual similarity (STS). By incorporating an attention mechanism to better capture inter-sentence interactions and using strong phrase-level word embeddings, their model achieved state-of-the-art results on the SemEval-2016 STS task, outperforming previous winning systems without relying on traditional feature engineering.
What Problem Does This Paper Address?
Measuring how semantically similar two pieces of text are is a fundamental challenge in natural language processing, crucial for tasks like paraphrase detection, question answering, and search ranking. While neural networks have shown promise, many existing models treat input sentences in isolation, potentially missing crucial contextual relationships between them. This paper aims to bridge this gap by developing a model that explicitly models these inter-sentence interactions to improve STS performance.
Methods
The core of the proposed system is an extension of the Multi-Perspective Convolutional Neural Network (MPCNN) model. The key innovations are:
- Attention-Based Input Interaction Layer: This new layer is introduced as the first step in processing input sentences. It uses an attention mechanism, inspired by recent work in neural machine translation, to re-weight word embeddings. Specifically, it calculates pairwise similarity scores between words of the two input sentences using cosine distance. These scores are then used to generate attention weights, which highlight words in one sentence that are more relevant to the other. This effectively creates "re-weighted" word embeddings that capture inter-sentence context before feeding them into the CNN.
- Multi-Perspective Convolutional Neural Network (MPCNN): This forms the base of their model. It uses multiple types of convolutional filters (holistic and per-dimension) with various window sizes and pooling strategies (max, min, mean) to capture information at different granularities and n-gram lengths within each sentence.
- Structured Similarity Measurement Layer: This layer compares the representations generated by the MPCNN for each sentence, considering local regions of these representations based on shared filter types, window sizes, and pooling strategies.
- PARAGRAM-PHRASE Word Embeddings: The authors leverage pre-trained word embeddings trained on phrase pairs from the Paraphrase Database (PPDB). These embeddings are noted for their strong performance, particularly when compared to word-level embeddings.
Notably, the authors explicitly state they do not use sparse features, syntactic parsers, or external resources like WordNet, focusing solely on the neural network architecture and pre-trained embeddings.
Datasets
- Training Data: The model was trained on data from previous SemEval STS competitions (STS2012-2015), totaling over 14,000 sentence pairs.
- Evaluation Data:
- SemEval-2016 STS Task 1: The primary evaluation was on the official test sets of the SemEval-2016 STS competition, which consists of five datasets from different domains (answer-answer, headlines, plagiarism, postediting, question-question), totaling 1,186 sentence pairs.
- STS2015 Test Sets: An ablation study was conducted on the STS2015 test data (3,000 sentence pairs) to evaluate the impact of different components.
Key Insights and Ideas
- Attention for Inter-Sentence Context is Crucial: The most significant insight is that explicitly modeling the interaction between sentences, rather than just processing them independently, leads to substantial improvements in STS performance. The attention mechanism effectively guides the model to focus on relevant word alignments.
- Phrase-Level Embeddings Outperform Word-Level: The use of PARAGRAM-PHRASE embeddings, trained on phrase pairs, proved highly beneficial, outperforming both GloVe and PARAGRAM-SL999 (word-pair trained) embeddings. This suggests that richer, phrase-level semantic representations are more effective for STS.
- Simplicity Can Be Powerful: The authors demonstrate that a well-designed neural architecture combined with strong embeddings can achieve top performance without relying on traditional, hand-crafted features or external linguistic resources.
Key Results
- SemEval-2016 STS Task 1: The proposed system achieved strong results on the SemEval-2016 STS task. While specific rankings aren't detailed in the provided text, the paper states their "final model outperforms the winning entry in STS2015." Table 2 shows their three submission runs achieved Pearson's r scores ranging from 0.7111 to 0.7420.
- Ablation Study (STS2015):
- Removing the attention layer resulted in a significant drop in performance (0.8040 to 0.7948 Pearson's r).
- Replacing PARAGRAM-PHRASE embeddings with GloVe led to a notable decrease (0.8040 to 0.7622).
- Replacing PARAGRAM-PHRASE with PARAGRAM-SL999 also showed a performance drop (0.8040 to 0.7721).
- Their full system's performance (0.8040) was comparable to, and slightly better than, the winning system of STS2015 (0.8015).
Limitations and Future Work
- Limitations: The paper does not explicitly detail limitations. However, it's implied that while their attention mechanism is simpler and more efficient than some other attention models, it still adds computational complexity compared to a non-attentional model. The reliance on pre-trained embeddings means performance is tied to the quality and coverage of those embeddings.
- Future Work: Not explicitly discussed in the paper, but the success of the attention mechanism and phrase-level embeddings suggests further exploration in these areas for other NLP tasks. The authors' focus on a unified neural approach also points towards continued research in end-to-end deep learning for semantic understanding.
Summary updated