Probing Neural Network Comprehension of Natural Language Arguments

Timothy Niven, Hung-Yu KaoACL 2019 · DOI

Introduction

Recent advancements in Natural Language Processing, particularly the advent of large-scale language models like BERT, have led to near-human performance on complex reasoning tasks. On the Argument Reasoning Comprehension Task (ARCT), BERT achieved a maximum accuracy of 77%, trailing the human baseline by only 3 percentage points. However, this paper demonstrates that such performance is not indicative of true logical comprehension but is instead the result of exploiting spurious statistical cues within the dataset.

The Argument Reasoning Comprehension Task (ARCT)

In the field of Argument Mining, understanding the relationship between claims and reasons is paramount. A critical component of this is the warrant: the implicit bridge of world knowledge that justifies why a specific reason supports a claim. For example, if the claim is "You should take an umbrella" and the reason is "It is raining," the warrant is "It is bad to get wet."

ARCT tests a model's ability to select the correct warrant from two options. An example of this task structure is shown below:

An example from the ARCT dataset illustrating the relationship between a claim, a reason, and the warrants (correct and alternative).

Methodology: Probing and Adversarial Testing

The authors employed three primary methods to investigate the validity of BERT's performance:

  1. Probing Experiments: Models were trained on partial versions of the data. For instance, providing the model with only the warrants while removing the claim and the reason entirely. If a model can still pick the right answer, it is clearly relying on "leaks" or artifacts in the warrants themselves.
  2. Statistical Cue Analysis: The authors calculated the productivity (how often a word predicts the correct answer) and coverage (how often that word appears). This identified specific "cue words" that models were likely over-fitting on.
  3. Adversarial Dataset Construction: To neutralize these cues, the authors created a "balanced" version of the dataset. For every original data point, they added a twin where the claim was negated and the labels were inverted. This ensured that any statistical cue present in a "correct" warrant would now appear exactly as often in an "incorrect" warrant.

Key Results: The Collapse of Reasoning

The results of the probing experiments were telling. As shown in the table below, BERT achieved a maximum accuracy of 71.2% when provided with only the warrants (BERT (W)), proving that the vast majority of its performance was derived from artifacts in the warrants alone.

Performance of BERT and baseline models on the ARCT test set with partial inputs. BERT (W) represents the model trained only on warrants.

When tested on the balanced adversarial dataset, BERT’s performance collapsed. Accuracy dropped to a maximum of 53%, with a mean and median of 50%—exactly what would be expected from random guessing. Standard models like Bag-of-Vectors (BoV) and BiLSTMs also failed on the adversarial set, confirming that the reliance on spurious cues was a universal problem across different architectures.

Discussion: The "Not" Heuristic

The most significant discovery was the power of the word "not." In the original ARCT dataset, if a warrant contained the word "not," it was the correct answer 61% of the time. BERT was essentially learning a "choose the sentence with 'not'" strategy rather than a logical reasoning strategy. Because BERT is a "strong learner" of noise, it is exceptionally good at finding even subtle statistical patterns. If a dataset has any unintended bias, BERT will find it and exploit it, leading to high scores that mask a total lack of conceptual understanding.

Conclusion

The primary takeaway is a call to action for the NLP community: high performance on a benchmark does not equal comprehension. The authors demonstrate that for tasks involving binary choice, the dataset must be mirrored to ensure models cannot rely on the distribution of words within the choices. Future work on ARCT must use the adversarial standard and likely incorporate external world knowledge to move beyond simple linguistic pattern matching.

Summary updated

BibTeX
@inproceedings{niven2019probing,
  author = {Timothy Niven and Hung-Yu Kao},
  title = {{Probing Neural Network Comprehension of Natural Language Arguments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1459},
}
Figure