Neural Conversational QA: Learning to Reason vs Exploiting Patterns

Nikhil Verma, Abhishek Sharma, Dhiraj Madan, Danish Contractor, Harshit Kumar, Sachindra JoshiEMNLP 2020 · OpenReview

Abstract: Nikhil Verma, Abhishek Sharma, Dhiraj Madan, Danish Contractor, Harshit Kumar, Sachindra Joshi. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

Researchers have discovered that top-performing neural models for conversational Question Answering (QA) are often "cheating" by exploiting spurious statistical patterns in datasets rather than actually reasoning through rules. By identifying these shortcuts and introducing a more robust version of the ShARC dataset (ShARC-Mod), the authors demonstrate a significant performance drop in state-of-the-art models, highlighting the need for more rigorous evaluation in conversational AI.

Problem Definition

In Conversational Question Answering (CQA), a system must navigate a dialogue by interpreting a set of rules (e.g., a policy document), looking at a user's specific scenario, and deciding whether to give a final "Yes/No" answer or ask a follow-up question to clarify missing information.

An example of the ShARC task structure, involving rule text, user scenario, and subsequent dialogue turns.

The ShARC (Shaping Answers with Rules through Conversation) dataset is the standard benchmark for this task. While neural models like BERT have achieved high scores on ShARC, the authors suspected these models weren't actually "reasoning" through the logic of the rules. Instead, they hypothesized that the models were picking up on "spurious patterns"—statistical artifacts of how the dataset was constructed that allow a model to guess the right answer without understanding the text.

Methodology

The authors took a multi-pronged approach to expose these shortcuts:

  1. Pattern Identification: They manually and statistically analyzed the ShARC corpus to find specific correlations that a model could exploit.
  2. The Heuristic Baseline: They built a simple, rule-based program (no machine learning involved) that uses basic heuristics—like checking if the scenario is empty or simply asking the next available rule clause as a question—to see how well a "brainless" script could perform.
  3. Dataset Perturbation: They created a "History-Shuffled" version of the evaluation set. If a model relies on the order of previous answers rather than logic, shuffling that history should break its performance.
  4. Metric Reformulation: They introduced BLEU-P (Penalized). The original evaluation didn't penalize models that failed to ask a follow-up question when one was required; BLEU-P fixes this "free pass."
  5. Dataset Mitigation: They developed ShARC-Mod, a modified version of the original dataset where these spurious patterns are diluted or removed through automated text transformations.

Identifying Spurious Patterns

The authors uncovered four primary "cheats" that models were using to inflate their performance:

  • The "Last Answer" Shortcut: In 74.6% of cases, the final answer to the user's question is identical to the user's last response in the dialogue history. A model can just "copy-paste" the last answer and be right most of the time.
  • The "Sequential" Shortcut: Rule clauses in ShARC are usually separated by asterisks. In 62.8% of cases, the "correct" follow-up question is simply the very next clause in the text that hasn't been asked yet. Models don't need to understand which clause is relevant; they just need to know which one is next.
  • The "Empty Scenario" Shortcut: If the user hasn't provided background info (scenario) and there's no history, there is a 66.67% chance the answer is "Irrelevant."
  • Turn Count Correlation: The likelihood of a model needing to ask a follow-up question decreases predictably as the number of turns increases, simply because there are a finite number of rules to check.

Experimental Results

The results confirm that neural models rely heavily on these shortcuts:

  • Heuristics vs. AI: The simple, hand-coded heuristic program achieved a 63.74% micro-accuracy, remarkably close to BERT-QA’s 68.63%. This proves that a significant portion of the task can be solved without any actual "understanding."
Comparison of model performance on the original ShARC dataset, highlighting the competitive performance of simple heuristics.
  • Performance Collapse: When tested on the "History-Shuffled" dataset, neural models saw their accuracy drop by an average of 8.16%, while the heuristic model remained stable. This confirms the models were over-relying on the sequence of the dialogue.
  • ShARC-Mod is Harder: On the new ShARC-Mod dataset, models performed consistently across both original and perturbed sets, but at a lower overall accuracy. This suggests ShARC-Mod is a more "honest" test of reasoning.
  • Metric Inflation: The authors showed that official BLEU scores were overestimating model quality by roughly 16% because they didn't account for valid alternative follow-up questions.

Conclusion and Future Work

The findings suggest that current SOTA models for conversational QA are often exploiting dataset artifacts rather than learning to reason. While ShARC-Mod provides a more robust benchmark, the authors acknowledge that automated perturbations can occasionally result in "unnatural" dialogue flows.

For future work, they recommend that dataset creators intentionally introduce perturbations during the data collection phase to prevent these patterns from forming. They also urge model developers to use "probing" (testing on adversarial or perturbed inputs) early in the design process to ensure their models are truly reasoning rather than just memorizing statistical noise.

Summary updated

BibTeX
@inproceedings{verma2020neural,
  author = {Nikhil Verma and Abhishek Sharma and Dhiraj Madan and Danish Contractor and Harshit Kumar and Sachindra Joshi},
  title = {{Neural Conversational QA: Learning to Reason vs Exploiting Patterns}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=mWrzsEsEOP},
}
Figure