Rule-based vs. Neural Net Approaches to Semantic Textual Similarity

Linrui Zhang, Dan MoldovanCOLING 2018

TL;DR

This paper introduces an attention-based Bidirectional Long Short-Term Memory (Bi-LSTM) neural network for Semantic Textual Similarity (STS) that achieves competitive results without relying on extensive linguistic features or external resources. By comparing its performance against traditional rule-based systems on the SemEval 2012 benchmark, the authors highlight the strengths and limitations of both approaches.

What Problem Does This Paper Address?

Semantic Textual Similarity (STS) is the task of determining how similar the meanings of two sentences are. Traditionally, STS systems have heavily relied on hand-crafted linguistic features and external knowledge bases like WordNet and Wikipedia. While these rule-based systems have achieved good performance, they require significant manual effort and domain expertise. This paper aims to demonstrate that end-to-end neural network models, specifically an attention-based Bi-LSTM, can achieve comparable or even superior results without this heavy reliance on linguistic resources, thus simplifying the development process and potentially improving robustness.

Methods

The authors propose an end-to-end neural network model based on an attention-based Bi-LSTM. The core components are:

  • Pre-processing: Sentences are tokenized, punctuation is removed, and all tokens are lowercased. Word embeddings (GloVe) are used, with non-existing words mapped to zero vectors. An additional flag is added to word embeddings to indicate if a word appears in both sentences. Sentences are then padded to a uniform length.
  • Attention-based Bi-LSTM: This architecture processes word embeddings to generate sentence-level representations. A Bi-LSTM is used to capture both forward and backward sequential information within sentences. An attention mechanism is applied to weigh the importance of different words in a sentence, allowing the model to focus on more informative parts.
  • Semantic Difference Vector: Similar to previous work, the semantic difference between the two sentence representations is computed by concatenating the element-wise absolute difference and element-wise multiplication of the sentence vectors.
  • Comparator: A two-layer fully connected neural network with a softmax output layer takes the semantic difference vector and generates a probabilistic distribution over similarity labels. This distribution is then converted into a continuous similarity score.

Datasets

The system was evaluated on the corpora from the SemEval 2012 Task 06: A Pilot on Semantic Textual Similarity. These include:

  • MSRvid: Short sentences for video descriptions.
  • MSRpar: Long sentences of paraphrases.
  • SMTeuroparl: Output of machine translation systems and reference translations.
  • OnWN: OntoNotes and WordNet glosses (surprise data).
  • SMTnews: Output of machine translation systems in the news domain and gold translations (surprise data).

For the surprise data (OnWN and SMTnews), the authors used training data from MSRpar and SMTeuroparl, respectively, due to similarities in sentence characteristics (long/hard to parse for MSRpar, ungrammatical for SMTeuroparl).

Key Insights and Ideas

  • Neural Networks Can Compete Without Heavy Feature Engineering: The paper strongly suggests that well-designed neural network models can achieve competitive STS performance without the extensive manual feature engineering and reliance on external linguistic resources that characterize traditional rule-based systems.
  • Attention is Crucial for Long Sentences: The inclusion of attention mechanisms significantly improves the model's ability to handle long sentences by allowing it to focus on the most relevant words and ignore less important ones, mitigating the "error propagation" issue common in feature-extraction-heavy systems.
  • Bi-LSTM Robustness for Ungrammatical Text: The Bi-LSTM architecture, by processing sentences in both forward and backward directions, demonstrates greater robustness in handling ungrammatical or unstructured sentences compared to a regular LSTM.
  • Rule-Based Systems Still Hold an Edge (Slightly): Despite the advancements in neural networks, the overall performance of rule-based models on the benchmark was still slightly better, particularly on new domain datasets. This highlights the current limitations of neural networks in generalizing to unseen domains without sufficient training data.

Key Results

The paper presents Pearson correlation coefficients as the evaluation metric.

  • The attention-based Bi-LSTM model achieved strong results, outperforming the regular LSTM model, especially when attention was applied.
  • On the MSRpar corpus (long sentences), the attention-based Bi-LSTM model beat the third-ranked performer and ranked second and third on the SMTeuroparl corpus.
  • The rule-based systems (Bär et al., 2012; Šarić et al., 2012; Banea et al., 2012) generally showed slightly higher correlation coefficients across most datasets compared to the neural network models.
  • For the surprise datasets (OnWN, SMTnews), rule-based models showed a significant advantage, underscoring their better generalization capabilities when training data is scarce.

Limitations and Future Work

  • Overall Performance Gap: The authors acknowledge that rule-based models still slightly outperform their neural network approach on average.
  • Generalization to New Domains: Neural network models, being supervised, struggle with generalization to new domains with limited training data, whereas rule-based systems are more robust in such scenarios due to their reliance on word pairings and linguistic resources.
  • Future Work: The paper implicitly suggests further research into improving the generalization capabilities of neural network models for STS, potentially by exploring more advanced architectures, transfer learning techniques, or incorporating limited external knowledge in a more integrated way.

Summary updated

BibTeX
@inproceedings{zhang2018rule,
  author = {Linrui Zhang and Dan Moldovan},
  title = {{Rule-based vs. Neural Net Approaches to Semantic Textual Similarity}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure