Abstract: Nikhil Verma, Abhishek Sharma, Dhiraj Madan, Danish Contractor, Harshit Kumar, Sachindra Joshi. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
Researchers have discovered that top-performing neural models for conversational Question Answering (QA) are often "cheating" by exploiting spurious statistical patterns in datasets rather than actually reasoning through rules. By identifying these shortcuts and introducing a more robust version of the ShARC dataset (ShARC-Mod), the authors demonstrate a significant performance drop in state-of-the-art models, highlighting the need for more rigorous evaluation in conversational AI.
Problem Definition
In Conversational Question Answering (CQA), a system must navigate a dialogue by interpreting a set of rules (e.g., a policy document), looking at a user's specific scenario, and deciding whether to give a final "Yes/No" answer or ask a follow-up question to clarify missing information.
The ShARC (Shaping Answers with Rules through Conversation) dataset is the standard benchmark for this task. While neural models like BERT have achieved high scores on ShARC, the authors suspected these models weren't actually "reasoning" through the logic of the rules. Instead, they hypothesized that the models were picking up on "spurious patterns"—statistical artifacts of how the dataset was constructed that allow a model to guess the right answer without understanding the text.
Methodology
The authors took a multi-pronged approach to expose these shortcuts:
- Pattern Identification: They manually and statistically analyzed the ShARC corpus to find specific correlations that a model could exploit.
- The Heuristic Baseline: They built a simple, rule-based program (no machine learning involved) that uses basic heuristics—like checking if the scenario is empty or simply asking the next available rule clause as a question—to see how well a "brainless" script could perform.
- Dataset Perturbation: They created a "History-Shuffled" version of the evaluation set. If a model relies on the order of previous answers rather than logic, shuffling that history should break its performance.
- Metric Reformulation: They introduced BLEU-P (Penalized). The original evaluation didn't penalize models that failed to ask a follow-up question when one was required; BLEU-P fixes this "free pass."
- Dataset Mitigation: They developed ShARC-Mod, a modified version of the original dataset where these spurious patterns are diluted or removed through automated text transformations.
Identifying Spurious Patterns
The authors uncovered four primary "cheats" that models were using to inflate their performance:
- The "Last Answer" Shortcut: In 74.6% of cases, the final answer to the user's question is identical to the user's last response in the dialogue history. A model can just "copy-paste" the last answer and be right most of the time.
- The "Sequential" Shortcut: Rule clauses in ShARC are usually separated by asterisks. In 62.8% of cases, the "correct" follow-up question is simply the very next clause in the text that hasn't been asked yet. Models don't need to understand which clause is relevant; they just need to know which one is next.
- The "Empty Scenario" Shortcut: If the user hasn't provided background info (scenario) and there's no history, there is a 66.67% chance the answer is "Irrelevant."
- Turn Count Correlation: The likelihood of a model needing to ask a follow-up question decreases predictably as the number of turns increases, simply because there are a finite number of rules to check.
Experimental Results
The results confirm that neural models rely heavily on these shortcuts:
- Heuristics vs. AI: The simple, hand-coded heuristic program achieved a 63.74% micro-accuracy, remarkably close to BERT-QA’s 68.63%. This proves that a significant portion of the task can be solved without any actual "understanding."
- Performance Collapse: When tested on the "History-Shuffled" dataset, neural models saw their accuracy drop by an average of 8.16%, while the heuristic model remained stable. This confirms the models were over-relying on the sequence of the dialogue.
- ShARC-Mod is Harder: On the new ShARC-Mod dataset, models performed consistently across both original and perturbed sets, but at a lower overall accuracy. This suggests ShARC-Mod is a more "honest" test of reasoning.
- Metric Inflation: The authors showed that official BLEU scores were overestimating model quality by roughly 16% because they didn't account for valid alternative follow-up questions.
Conclusion and Future Work
The findings suggest that current SOTA models for conversational QA are often exploiting dataset artifacts rather than learning to reason. While ShARC-Mod provides a more robust benchmark, the authors acknowledge that automated perturbations can occasionally result in "unnatural" dialogue flows.
For future work, they recommend that dataset creators intentionally introduce perturbations during the data collection phase to prevent these patterns from forming. They also urge model developers to use "probing" (testing on adversarial or perturbed inputs) early in the design process to ensure their models are truly reasoning rather than just memorizing statistical noise.
Summary updated