A Pipeline for Generating, Annotating and Employing Synthetic Data for Real World Question Answering

Matt Maufe, James Ravenscroft, Rob Procter, Maria LiakataEMNLP 2022

TL;DR

This paper introduces a flexible pipeline for generating synthetic question-answering (QA) data, which is crucial for improving QA model performance on domain-specific tasks where large, manually annotated datasets are scarce. By leveraging existing language models and a novel annotation tool, they demonstrate significant performance gains, particularly for unanswerable questions, by fine-tuning models on this synthetic data.

What Problem Does This Paper Address?

State-of-the-art Question Answering (QA) models, often trained on general domain datasets like Wikipedia, struggle when applied to specialized domains (e.g., business news) due to a lack of relevant training data. Manually creating these domain-specific datasets is expensive and time-consuming, requiring skilled annotators. This paper addresses the challenge of acquiring sufficient high-quality, domain-specific training data for QA tasks by proposing a method to generate and annotate synthetic data.

Methods

The core of the proposed solution is a modular, architecture-agnostic pipeline. The process begins with a question generation (QG) model that takes unlabelled domain-specific documents and produces synthetic QA pairs. This QG process involves two T5 models: one for answer candidate selection and another for generating questions conditioned on the answer and document.

To ensure the quality of the generated data, a grammaticality model (a pre-trained BERT model fine-tuned on the CoLA dataset) is used to filter out ungrammatical questions. The pipeline also includes a web-based annotation tool that allows human annotators to label various aspects of the synthetic data, such as suitability, naturalness, and correctness. This annotated data is then used to fine-tune both the grammaticality and QA models. The authors also explore techniques for improving the detection of unanswerable questions, including using "source markers" and multi-task learning.

Datasets

  • Domain-General Data: SQUAD 2.0 (for initial QA model fine-tuning) and CoLA (for grammaticality model training).
  • Domain-Specific Data: A proprietary knowledge base focused on business news, which was used to create the "SYFTER" dataset.

Key Insights and Ideas

  • Synthetic Data for Domain Adaptation: Synthetic data, even when generated from general-purpose models, can significantly improve QA performance on domain-specific tasks, bridging the gap where real-world data is limited.
  • Grammaticality Matters: Filtering synthetic questions for grammaticality is crucial for improving downstream QA performance, suggesting that natural language fluency is a key factor for effective QA.
  • Annotation Tool Value: A well-designed annotation tool and clear guidelines are essential for obtaining high-quality labels from human annotators, even for synthetic data.
  • Unanswerable Questions are Key: The ability to correctly identify unanswerable questions is critical for robust QA systems, and synthetic data can be leveraged to improve this capability.
  • Multi-task Learning Benefits: Combining QA with sequence classification for answerability detection can lead to improved overall performance.

Key Results

  • Fine-tuning a QA model on their domain-specific synthetic dataset ("SYFTER") improved performance by 8.75 F1 compared to a model trained solely on domain-general data.
  • The grammaticality model achieved strong results, with the domain-specific setting performing better than domain-general or combined settings.
  • Human evaluation showed that a high percentage of the generated synthetic QA pairs were of good quality (e.g., 69.7% were suitable and had correct answers).
  • Combining domain-general (SQUAD 2.0) and domain-specific (SYFTER) data for QA model training yielded the best overall performance, achieving 64.96 EM and 74.00 F1 on the SYFTER test set.

Limitations and Future Work

  • Adversarial Questions: The current system does not consider "adversarial questions," which could limit its robustness against such inputs.
  • Entity-Centric Questions: The generated questions tend to focus on entities, potentially limiting applicability to questions about relationships between entities. Future work could explore tools like AdaTest to improve this.
  • Human Annotation Still Needed: While the synthetic data is of high quality, human annotation is still required to filter and label it, particularly for identifying unanswerable questions. The authors suggest future work could explore generating deliberately unanswerable examples.
  • Bias in Data: Bias can be carried over from the generative models and human annotators. Future work should investigate debiasing techniques.
  • Grammaticality as a Secondary Objective: The authors suggest that training the QG and answer selection models with grammaticality as a secondary objective could further improve input quality.

Summary updated

BibTeX
@inproceedings{maufe2022pipeline,
  author = {Matt Maufe and James Ravenscroft and Rob Procter and Maria Liakata},
  title = {{A Pipeline for Generating, Annotating and Employing Synthetic Data for Real World Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
}
Figure