Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation

Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa

TL;DR

This paper introduces VQAG, a variational framework for generating diverse synthetic question-answer pairs to improve the robustness of reading comprehension models. By using independent latent variables to control the diversity of both answers and questions, the authors demonstrate that diversity-promoting data augmentation significantly boosts performance across 12 different "challenge sets" without requiring prior knowledge of the target distribution.

What Problem Does This Paper Address?

Modern Question Answering (QA) models often achieve human-level performance on standard benchmarks like SQuAD. However, they are notoriously brittle: their performance plummets when faced with "challenge sets"—test data that involves paraphrasing, adversarial noise, or shifts in domain (e.g., moving from Wikipedia to news articles).

Existing solutions typically rely on targeted data augmentation. For instance, if you know your model will face paraphrased questions, you train it on synthetic paraphrases. The major gap here is that in real-world applications, we rarely know what specific "traps" or distribution shifts the model will encounter. This paper addresses the need for a target-agnostic method to improve robustness by tackling the underlying sparsity and lack of diversity in original training datasets.

Methods

The authors propose a Variational Question-Answer pair Generation (VQAG) model. Unlike standard generative models that aim for a single "best" output, VQAG is designed to produce a wide variety of plausible QA pairs from a single context.

Key Innovations:

  • Dual Latent Variables: The model introduces two independent latent random variables: $z$ to diversify Answer Extraction (AE) and $y$ to diversify Question Generation (QG). This captures the one-to-many nature of QA—a single paragraph can contain many "question-worthy" answers, and each answer can be queried by many different questions.
  • Explicit KL Control: To prevent "posterior collapse" (a common VAE issue where the model ignores the latent variables and produces repetitive outputs), the authors use a modified objective function. By tuning hyperparameters $C_a$ and $C_q$, they can explicitly control the amount of information the latent variables contribute, effectively acting as a "diversity knob."
  • Heuristic Filtering: Since VAEs can sometimes produce noisy or ungrammatical text, the authors apply simple heuristic filters (e.g., length constraints, presence of interrogative words) to ensure the synthetic data remains useful for training.
Figure 1: Overview of the model architecture. The la- tent variables z and y are sampled from the posteriors when computing the variational lower bound and from the priors during generation. See §3.3 for the details.

Datasets

  • Source Dataset: SQuAD v1.1 served as the primary source for training the VQAG model and the baseline QA models.
  • Evaluation (Challenge Sets): The authors evaluated robustness on 12 distinct challenge sets, including:
  • Domain shifts: NewsQA, Natural Questions (NQ).
  • Adversarial/Linguistic challenges: AddSent, AddOneSent, Quoref (coreference resolution), and Implications.
  • Robustness to noise: NoiseQA (MT, ASR, and Keyboard noise).
  • Paraphrasing: Adversarial and non-adversarial paraphrased sets.

Key Insights and Ideas

  • Diversity Over Quality: While many QAG research efforts focus on making synthetic questions look as "human-like" as possible, this paper argues that diversity is more important for robustness. Even if synthetic pairs are slightly noisy, the broader coverage of the input space helps the downstream QA model generalize better.
  • Target-Agnostic Robustness: VQAG improved performance across nearly all challenge sets simultaneously. This suggests that increasing training set diversity is a viable "general-purpose" defense against distribution shift.
  • Question Type Distribution: Standard datasets like SQuAD are heavily biased toward "what" questions. VQAG naturally generates a more balanced distribution, including significantly more "how" and "which" questions, which forces the model to learn more complex reasoning patterns.
Table 3: Heatmap of extracted answer spans and generated samples using our model. The darker the color is, the more often the word is extracted. The phrases surrounded by black boxes are the ground truth answers in SQuAD.

Key Results

The VQAG-augmented models showed consistent improvements over the standard BERT baseline:

  • General Robustness: VQAG improved the F1 score on all 12 challenge sets. For example, on the Adversarial Paraphrased (APara) set, the F1 score jumped from 50.2 to 59.0.
  • In-Distribution Gains: Unlike some augmentation methods that trade off standard accuracy for robustness, VQAG also improved (or maintained) performance on the original SQuAD test set (increasing from 83.5 to 84.5 F1).
  • Ensemble Power: Combining models trained on different synthetic datasets (e.g., {+Sem,+Info,+V}) yielded the best overall results, reaching an F1 of 85.8 on SQuAD and setting new high-water marks for robustness across the board.
Main results table comparing VQAG against other augmentation methods across 12 challenge sets.

Limitations and Future Work

  • Grammatical Noise: Human evaluation revealed that about 25% of generated questions were not fully understandable, and 30% of answers were technically incorrect for the generated question. While the diversity helped, cleaner generation could potentially yield even higher gains.
  • Unanswerable Questions: The current model focuses on extractive QA where an answer always exists. The authors note that the lack of "answerability" in synthetic data might hinder performance on datasets like SQuAD 2.0, which include unanswerable questions.
  • Future Directions: The authors suggest applying VQAG to unlabeled documents from diverse domains (like news or medical texts) to further bridge the gap between training and real-world application environments.

Summary updated

Abstract

Question answering (QA) models for reading comprehension have achieved human-level accuracy on in-distribution test sets. However, they have been demonstrated to lack robustness to challenge sets, whose distribution is different from that of training sets. Existing data augmentation methods mitigate this problem by simply augmenting training sets with synthetic examples sampled from the same distribution as the challenge sets. However, these methods assume that the distribution of a challenge set is known a priori, making them less applicable to unseen challenge sets. In this study, we focus on question-answer pair generation (QAG) to mitigate this problem. While most existing QAG methods aim to improve the quality of synthetic examples, we conjecture that diversity-promoting QAG can mitigate the sparsity of training sets and lead to better robustness. We present a variational QAG model that generates multiple diverse QA pairs from a paragraph. Our experiments show that our method can improve the accuracy of 12 challenge sets, as well as the in-distribution accuracy. Our code and data are available at https://github.com/KazutoshiShinoda/VQAG.
BibTeX
@inproceedings{shinoda2021improving,
  author = {Kazutoshi Shinoda and Saku Sugawara and Akiko Aizawa},
  title = {{Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure