Quality > Quantity: Synthetic Corpora from Foundation Models for Closed-Domain Extractive Question Answering

Saptarshi Sengupta, Connor Heaton, Shreya Ghosh, Preslav Nakov, Prasenjit MitraSubmission to EMNLP 2023 · Rejected

TL;DR

TOP-Training is a new pretraining paradigm that uses Large Language Models (LLMs) to generate high-quality synthetic text tailored to specific, small medical datasets. By pretraining smaller bidirectional encoders (like BERT) on this "target-oriented" data, researchers achieved state-of-the-art results on medical extractive question-answering, outperforming models trained on massive real-world datasets.

What Problem Does This Paper Address?

Medical Extractive Question Answering (Medical-EQA)—where a system must find the exact span of text in a medical record that answers a query—faces three massive hurdles:

  1. Domain Specificity: Medical text is full of rare jargon that general-purpose AI models simply don't understand.
  2. Hallucination Risks: Modern generative LLMs (like GPT-4) are prone to "hallucinating" facts. In medicine, where accuracy is a matter of life and death, extractive models (which only select existing text) are safer, but they require significant training data.
  3. Data Scarcity & Privacy: Expert-labeled medical data is incredibly expensive to produce. Furthermore, privacy regulations (like HIPAA) often prevent researchers from accessing the large-scale, unlabeled Electronic Health Record (EHR) data typically needed for domain adaptation.

The authors ask: How can we train highly accurate medical models when we have very little data and no access to large-scale clinical databases?

Methods

The authors propose TOP-Training (Target-Oriented Pretraining), a three-phase pipeline:

  1. Synthetic Corpora Generation: Instead of searching for more real data, they make it. They extract medical entities from their small target dataset using Named Entity Recognition (NER). They then prompt a science-focused LLM (Galactica) to generate synthetic research articles or radiology reports containing those specific entities.
  2. In-Domain Pre-training: They take a standard bidirectional encoder (BERT or RoBERTa) and perform "extended pretraining" on this synthetic text. This familiarizes the model with the specific vocabulary and style of the target domain.
  3. Task-Specific Fine-tuning: Finally, the model undergoes two rounds of fine-tuning: first on a large general QA dataset (SQuAD) to learn the mechanics of extraction, and then on the tiny target medical dataset to master the specific task.

The "secret sauce" is the Fancy Prompting—carefully crafting prompts so the LLM generates text that mimics the specific genre of the target data (e.g., structured radiology reports vs. academic papers).

Datasets

The study evaluates the method on two primary benchmarks:

  • COVID-QA: 2,019 question-answer pairs sourced from CORD-19 research articles.
  • RadQA: 6,148 question-answer pairs based on radiology reports.
  • SQuAD (v1.1/v2.0): Used for intermediate fine-tuning to teach the model how to perform extractive QA.
  • Wikipedia: Used as a baseline "general" synthetic corpus for comparison.

Key Insights and Ideas

  • Quality > Quantity: Traditional domain adaptation relies on gigabytes of unlabeled text. TOP-Training shows that a tiny, highly relevant synthetic corpus (e.g., 67 MB) can outperform models trained on 94 GB of general medical text.
  • Target-Oriented vs. Domain-Oriented: Most researchers try to move models toward the "medical domain." This paper argues for moving models toward the target dataset. By using the specific entities found in the test set's domain to generate training data, the model learns the exact "neighborhood" of knowledge it needs.
  • Encoders Still Rule Extraction: While generative "decoder-only" models (like MedAlpaca) are popular, they struggle with the rigid requirements of extractive QA. Bidirectional encoders (BERT-style) remain superior for pinpointing answers within a text without hallucinating.
  • Style is a Signal: The model performed significantly better when the synthetic data matched the format of the target data (e.g., including "Findings" and "Impressions" sections for radiology tasks).

Key Results

  • New SOTA on COVID-QA: RoBERTa with TOP-Training achieved 41.51 Exact Match (EM) and 69.10 F1 score, surpassing the previous state-of-the-art model (LUKE), despite LUKE being trained on a 3.5-billion-word corpus.
  • Outperforming Real Data: On the RadQA dataset, TOP-Training outperformed RadLing, a model specifically trained on 500,000 real radiology reports, by 4.8% in F1 score.
  • Efficiency: The method is remarkably resource-efficient. It achieves top-tier results using a fraction of the data and training time compared to massive foundation models.
  • Zero-Shot Failure: The authors found that even specialized medical LLMs like MedLLaMA and MedAlpaca performed poorly (near 0% EM) on these extractive tasks without specific fine-tuning, highlighting the continued need for specialized encoder models.

Limitations and Future Work

  • Synthetic Hallucinations: While the final QA model is extractive (and thus safe), the synthetic data used for pretraining may contain LLM-generated hallucinations. The authors currently have no mechanism to filter these out.
  • Quality Control: The study measures success by downstream task performance but does not yet have a metric to evaluate the intrinsic "quality" or medical factualness of the generated synthetic corpora.
  • Privacy Nuance: While the method avoids using large private datasets, it still requires a "seed" of real data to extract entities. Future work could explore generating these entities synthetically as well to further enhance privacy.

Summary updated

Abstract

Domain adaptation, the process of training a model in one domain and applying it to another, has been extensively explored in machine learning. While training a domain-specific foundation model (FM) from scratch is an option, recent methods have focused on adapting pre-trained FMs for domain-specific tasks. However, our experiments reveal that either approach does not consistently achieve state-of-the-art (SOTA) results in the target domain. In this work, we study extractive question answering within closed domains and introduce the concept of targeted pre-training. This involves determining and generating relevant data to further pre-train our models, as opposed to the conventional philosophy of utilizing domain-specific FMs trained on a wide range of data. Our proposed framework uses Galactica to generate synthetic, ``targeted'' corpora that align with specific writing styles and topics, such as research papers and radiology reports. This process can be viewed as a form of knowledge distillation. We apply our method to two biomedical extractive question answering datasets, COVID-QA and RadQA, achieving a new benchmark on the former and demonstrating overall improvements on the latter. Code available upon publication.
BibTeX
@misc{sengupta2023quality,
  author = {Saptarshi Sengupta and Connor Heaton and Shreya Ghosh and Preslav Nakov and Prasenjit Mitra},
  title = {{Quality > Quantity: Synthetic Corpora from Foundation Models for Closed-Domain Extractive Question Answering}},
  year = {2023},
  eprint = {2310.16995},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.16995},
}
Figure