Quality > Quantity: Synthetic Corpora from Foundation Models for Closed-Domain Extractive Question Answering

Saptarshi Sengupta, Connor T. Heaton, Shreya Ghosh, Preslav Nakov, Prasenjit Mitra

Abstract

Domain adaptation, the process of training a model in one domain and applying it to another, has been extensively explored in machine learning. While training a domain-specific foundation model (FM) from scratch is an option, recent methods have focused on adapting pre-trained FMs for domain-specific tasks. However, our experiments reveal that either approach does not consistently achieve state-of-the-art (SOTA) results in the target domain. In this work, we study extractive question answering within closed domains and introduce the concept of targeted pre-training. This involves determining and generating relevant data to further pre-train our models, as opposed to the conventional philosophy of utilizing domain-specific FMs trained on a wide range of data. Our proposed framework uses Galactica to generate synthetic, "targeted" corpora that align with specific writing styles and topics, such as research papers and radiology reports. This process can be viewed as a form of knowledge distillation. We apply our method to two biomedical extractive question answering datasets, COVID-QA and RadQA, achieving a new benchmark on the former and demonstrating overall improvements on the latter.

Introduction

Medical Extractive Question Answering (Medical-EQA) is a critical task where AI systems must locate the exact span of text in a medical record that answers a specific query. Unlike generative models that may hallucinate, extractive models are safer for clinical applications because they are restricted to existing text. However, developing these models is hindered by three major obstacles: extreme domain specificity (jargon), the high cost of expert labeling, and data privacy restrictions (e.g., HIPAA) that limit access to large-scale Electronic Health Record (EHR) databases.

To address these challenges, we propose TOP-Training (Target-Oriented Pretraining). This paradigm shifts the focus from broad domain adaptation to "targeted" adaptation, using Large Language Models (LLMs) to generate high-quality synthetic text tailored to the specific style and vocabulary of a small target dataset.

Methodology

The TOP-Training framework consists of a three-phase pipeline designed to maximize the utility of limited domain-specific data.

The TOP-Training pipeline: Synthetic corpora generation, in-domain pre-training, and task-specific fine-tuning.

Phase 1: Synthetic Corpora Generation

Rather than searching for external real-world data, we generate a "targeted" corpus. We first extract medical entities from the small target dataset using Named Entity Recognition (NER). These entities are then used to prompt a science-focused LLM, such as Galactica, to generate synthetic documents (e.g., research articles or radiology reports) that incorporate these specific terms. This ensures the pre-training data is perfectly aligned with the "neighborhood" of knowledge required for the target task.

Phase 2: In-Domain Pre-training

We perform "extended pre-training" on a standard bidirectional encoder (like BERT or RoBERTa) using the generated synthetic corpora. This step allows the model to internalize the specific linguistic patterns and technical vocabulary of the target domain without requiring access to private clinical records.

Phase 3: Task-Specific Fine-tuning

The model undergoes a two-stage fine-tuning process. First, it is trained on a large-scale general QA dataset (SQuAD) to learn the fundamental mechanics of extractive question answering. Finally, it is fine-tuned on the small, expert-labeled target medical dataset to master the specific nuances of the domain.

Experimental Setup

We evaluated TOP-Training on two primary biomedical benchmarks:

  • COVID-QA: 2,019 QA pairs from CORD-19 research articles.
  • RadQA: 6,148 QA pairs based on radiology reports.

We compared our approach against models trained on massive real-world datasets, including LUKE (trained on 3.5 billion words) and RadLing (trained on 500,000 real radiology reports).

Results and Discussion

Our results demonstrate that the relevance of pre-training data is more important than its volume.

  • State-of-the-Art Performance: On COVID-QA, RoBERTa with TOP-Training achieved a new SOTA with 41.51 Exact Match (EM) and 69.10 F1 score, outperforming the previous best model (LUKE).
  • Synthetic vs. Real Data: On the RadQA dataset, TOP-Training outperformed RadLing by 4.8% in F1 score. This is significant because RadLing was trained on a massive corpus of real radiology reports, while TOP-Training used a much smaller synthetic corpus (approx. 67 MB).
  • Efficiency: TOP-Training achieves these results with a fraction of the data and computational resources required by massive foundation models.
  • Encoder Superiority: We found that specialized medical LLMs (like MedAlpaca) performed poorly on extractive tasks in a zero-shot setting, confirming that bidirectional encoders remain the optimal architecture for pinpointing answers within text.

Conclusion

TOP-Training provides a robust pathway for adapting AI to specialized, data-scarce domains like medicine. By leveraging LLMs to generate targeted synthetic corpora, we can bypass the hurdles of data privacy and scarcity while achieving superior performance. Future work will focus on improving the factual consistency of synthetic data and developing metrics to evaluate the intrinsic quality of generated medical corpora.

Summary updated

Abstract

Domain adaptation, the process of training a model in one domain and applying it to another, has been extensively explored in machine learning. While training a domain-specific foundation model (FM) from scratch is an option, recent methods have focused on adapting pre-trained FMs for domain-specific tasks. However, our experiments reveal that either approach does not consistently achieve state-of-the-art (SOTA) results in the target domain. In this work, we study extractive question answering within closed domains and introduce the concept of targeted pre-training. This involves determining and generating relevant data to further pre-train our models, as opposed to the conventional philosophy of utilizing domain-specific FMs trained on a wide range of data. Our proposed framework uses Galactica to generate synthetic, ``targeted'' corpora that align with specific writing styles and topics, such as research papers and radiology reports. This process can be viewed as a form of knowledge distillation. We apply our method to two biomedical extractive question answering datasets, COVID-QA and RadQA, achieving a new benchmark on the former and demonstrating overall improvements on the latter. Code available at https://github.com/saptarshi059/CDQA-v1-Targetted-PreTraining/tree/main.
BibTeX
@misc{sengupta2023quality,
  author = {Saptarshi Sengupta and Connor T. Heaton and Shreya Ghosh and Preslav Nakov and Prasenjit Mitra},
  title = {{Quality > Quantity: Synthetic Corpora from Foundation Models for Closed-Domain Extractive Question Answering}},
  year = {2023},
  eprint = {2310.16995},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.16995},
}
Figure