Improving Natural Language Inference with a Pretrained Parser

Deric Pang, Lucy H. Lin, Noah A. SmitharXiv 2019 · OpenReview

TL;DR

This paper introduces a novel method to inject syntactic information from dependency parsers into Natural Language Inference (NLI) models. By leveraging contextual token-level representations from a pretrained parser, the authors demonstrate consistent accuracy improvements across several strong NLI models and benchmark datasets, suggesting that explicit syntactic structure can enhance semantic understanding.

What Problem Does This Paper Address?

Natural Language Inference (NLI) tasks, which involve determining the relationship between two sentences (entailment, contradiction, or neutral), have seen significant progress with the advent of large pre-trained language models. However, these models often treat sentences as sequences of tokens without explicitly leveraging their underlying syntactic structure. While linguistic theory has long posited a strong connection between syntax and semantics, and parsing accuracy has dramatically improved, effectively integrating this syntactic knowledge into modern NLI models remains a challenge. This paper aims to bridge this gap by exploring how to best incorporate syntactic information from dependency parsers to boost NLI performance.

Methods

The core contribution of this paper is a method for extracting and integrating Syntactic Word Representations (SWRs) from a pretrained dependency parser into NLI models. SWRs are the contextual token-level vector representations produced by the parser's encoder. The authors propose two primary ways to leverage these SWRs:

  1. Late Fusion (+LF): This is the simpler approach where the SWRs are concatenated with the final contextual representations of the premise and hypothesis before they are fed into the model's final classification layer. This method is highly general and adds minimal parameters.
  2. Syntactic Attention (+SA): This method integrates SWRs into the attention mechanism of NLI models. Specifically, it modifies the dot product calculation used for attention weights by incorporating the SWRs of the premise and hypothesis tokens. This approach doesn't require learning new weights on the SWRs themselves.

Both methods are designed to be broadly applicable to various neural NLI architectures.

Datasets

The authors evaluate their methods on three standard NLI datasets:

  • SNLI (Stanford Natural Language Inference): A large dataset of 570k sentence pairs, with premises drawn from the Flickr 30k image caption dataset.
  • MNLI (Multi-Genre Natural Language Inference): A 433k sentence pair dataset covering ten different genres, split into matched (in-domain) and mismatched (cross-domain) test sets.
  • SciTail: A smaller dataset of 27k sentence pairs derived from science multiple-choice questions, notable for not having a contradiction label.

They also use the HANS (Heuristic Analysis for NLI Systems) dataset for analysis, which is designed to probe NLI models for reliance on specific syntactic heuristics.

Key Insights and Ideas

  • Syntax Matters, Even for Big Models: Even state-of-the-art models like BERT and MT-DNN, which are pre-trained on massive datasets, benefit from the explicit inclusion of syntactic information. This suggests that while large-scale pre-training captures a lot of linguistic knowledge, it doesn't fully internalize all aspects of syntactic structure relevant for NLI.
  • Two Integration Strategies Offer Different Trade-offs: Both Late Fusion (+LF) and Syntactic Attention (+SA) provide benefits, but their effectiveness can vary. +LF is more general and adds parameters, while +SA integrates directly into attention without adding parameters, potentially offering a more efficient way to leverage syntax.
  • Syntactic Attention is Sensitive to Noise: The +SA method, which directly uses parser outputs in attention calculations, is more susceptible to noise in the parser's representations. Ablation studies showed that replacing SWRs with random noise significantly degraded performance for +SA models, whereas +LF models were more robust.
  • Improved NLI Correlates with Better Heuristic Handling: The analysis on the HANS dataset reveals that models improved by SWRs also show better performance on HANS, particularly by balancing entailment and non-entailment predictions. This indicates that syntactic information helps models avoid relying on superficial heuristics and learn more robust semantic reasoning.

Key Results

  • Consistent Accuracy Gains: The proposed methods (+LF and +SA) consistently improved test accuracy across four baseline NLI models (Decomposable Attention, ESIM, BERT, MT-DNN) on the SNLI and SciTail datasets.
  • Mixed but Positive Results on MNLI: On the MNLI dataset, the results were more mixed. DA, BERT, and MT-DNN showed improvements on the matched subset, while DA and MT-DNN improved on the mismatched subset.
  • Specific Improvements: For example, DA+SA on SciTail saw a +3.1% accuracy increase, ESIM+SA on SciTail improved by +5.1%, BERT+LF on SciTail by +2.0%, and MT-DNN+LF on MNLI-mm by +0.5%.
  • Ablation Study Findings: Replacing SWRs with random noise in the +SA method led to significant accuracy drops (e.g., -7.6% on SNLI for DA+SAN), highlighting the importance of the parser's output.

Limitations and Future Work

  • Parser Dependence: The performance of the proposed methods is inherently tied to the accuracy of the pretrained dependency parser. While the paper uses a strong parser, errors in parsing could potentially hinder performance.
  • Sensitivity of +SA: The Syntactic Attention (+SA) method proved sensitive to noisy parser outputs, suggesting that more robust integration strategies might be needed for this approach.
  • Future Work: The authors suggest exploring other parser types (e.g., constituency parsers) and investigating how multi-head self-attention in models like BERT could be further leveraged for +SA variants. They also motivate future research into syntactically informed models for a deeper understanding of language.

Summary updated

Abstract

We introduce a novel approach to incorporate syntax into natural language inference (NLI) models. Our method uses contextual token-level vector representations from a pretrained dependency parser. Like other contextual embedders, our method is broadly applicable to any neural model. We experiment with four strong NLI models (decomposable attention model, ESIM, BERT, and MT-DNN), and show consistent benefit to accuracy across three NLI benchmarks.
BibTeX
@misc{pang2019improving,
  author = {Deric Pang and Lucy H. Lin and Noah A. Smith},
  title = {{Improving Natural Language Inference with a Pretrained Parser}},
  year = {2019},
  eprint = {1909.08217},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.08217},
}
Figure