Syntactic Data Augmentation Increases Robustness to Inference Heuristics

Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, Tal Linzen

Abstract

Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets. We hypothesize that this issue is not primarily caused by the pretrained model's limitations, but rather by the paucity of crowdsourced NLI examples that might convey the importance of syntactic structure at the fine-tuning stage. We explore several methods to augment standard training sets with syntactically informative examples, generated by applying syntactic transformations to sentences from the MNLI corpus. The best-performing augmentation method, subject/object inversion, improved BERT's accuracy on controlled examples that diagnose sensitivity to word order from 0.28 to 0.73, without affecting performance on the MNLI test set. This improvement generalized beyond the particular construction used for data augmentation, suggesting that augmentation causes BERT to recruit abstract syntactic representations.

Introduction

In Natural Language Inference (NLI), a model determines if a "premise" entails a "hypothesis." While models like BERT achieve human-level performance on standard datasets like MNLI, they are easily fooled by shallow heuristics. For instance, if the premise is "The lawyer saw the actor," these models often incorrectly conclude it entails "The actor saw the lawyer."

This behavior stems from a lexical overlap heuristic: models assume that high word-overlap between sentences automatically equals entailment. This paper investigates whether this is due to Representational Inadequacy (BERT lacks syntactic knowledge) or a Missed Connection (BERT has the knowledge but the training data doesn't incentivize using it).

Methods

The authors test these hypotheses using Syntactic Data Augmentation. They transformed existing MNLI examples into new training pairs using two primary methods:

  • Inversion: Swapping the subject and object (e.g., "The doctor called the patient" $\rightarrow$ "The patient called the doctor").
  • Passivization: Changing active voice to passive (e.g., "The doctor called the patient" $\rightarrow$ "The patient was called by the doctor").

Augmentation sizes varied from 101 to 1215 examples—a tiny fraction (approx. 0.1%) of the full MNLI dataset. Models were evaluated on the standard MNLI development set and the HANS (Heuristic Analysis for NLI Systems) challenge set, which specifically targets lexical overlap, subsequence, and constituent heuristics.

Key Results and Insights

The most successful strategy—Inversion with a transformed hypothesis—dramatically improved robustness. On HANS cases where the lexical overlap heuristic leads to an incorrect prediction (non-entailment), BERT's accuracy rose from 0.28 in the baseline to 0.73 with 1215 augmentation examples.

Accuracy on HANS lexical overlap cases across different augmentation strategies.

Crucially, this improvement generalized to complex structures not present in the augmentation data, such as relative clauses (0.33 $\rightarrow$ 0.83). This supports the "Missed Connection" hypothesis: BERT already possesses the necessary syntactic representations from pretraining, but requires a small "nudge" during fine-tuning to realize these representations are relevant for the NLI task. Notably, this robustness boost came with no "performance tax" on the standard MNLI set, where accuracy remained stable at 0.84.

However, the effectiveness of augmentation varied across different types of heuristics. While the lexical overlap heuristic was easily broken, the "subsequence" and "constituent" heuristics proved more stubborn.

Comparison of augmentation effects across lexical overlap, subsequence, and constituent heuristics.

Discussion and Limitations

Despite the success of inversion, models still struggled with passive voice, suggesting that BERT's pretraining might be representationally inadequate regarding passives. Furthermore, the modest improvements on subsequence and constituent heuristics indicate that broader, more diverse augmentation strategies are necessary to achieve full robustness.

In conclusion, the results demonstrate that even a minuscule amount of syntactically informative data can significantly shift a model's reliance from shallow heuristics to structural reasoning. Future work should focus on creating more diverse training sets that represent a wider range of linguistic phenomena beyond the simple examples typically generated by crowdworkers.

Summary updated

Abstract

Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets. We hypothesize that this issue is not primarily caused by the pretrained model's limitations, but rather by the paucity of crowdsourced NLI examples that might convey the importance of syntactic structure at the fine-tuning stage. We explore several methods to augment standard training sets with syntactically informative examples, generated by applying syntactic transformations to sentences from the MNLI corpus. The best-performing augmentation method, subject/object inversion, improved BERT's accuracy on controlled examples that diagnose sensitivity to word order from 0.28 to 0.73, without affecting performance on the MNLI test set. This improvement generalized beyond the particular construction used for data augmentation, suggesting that augmentation causes BERT to recruit abstract syntactic representations.
BibTeX
@inproceedings{min2020syntactic,
  author = {Junghyun Min and R. Thomas McCoy and Dipanjan Das and Emily Pitler and Tal Linzen},
  title = {{Syntactic Data Augmentation Increases Robustness to Inference Heuristics}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=6ItxF1jf5QK},
}
Figure