Open-Domain Question-Answering for COVID-19 and Other Emergent Domains

Sharon Levy, Kevin Mo, Wenhan Xiong, William Yang Wang

TL;DR

This paper presents an open-domain question-answering (QA) system specifically designed for the rapidly evolving COVID-19 domain. It tackles the challenge of limited domain-specific data by leveraging pre-trained language models and a multi-stage retrieval and re-ranking pipeline to deliver accurate and diverse answers from a large corpus of scientific literature.

What Problem Does This Paper Address?

The COVID-19 pandemic created an unprecedented surge of scientific information, often accompanied by misinformation. This created a critical need for a reliable way for users to access accurate, science-based answers to their questions. Traditional QA systems often struggle with emergent domains due to a lack of large, curated datasets and the dynamic nature of the information. This paper addresses the challenge of building an effective QA system for such a domain, where data is scarce and constantly changing, by adapting existing QA techniques.

Methods

The system employs a two-stage approach: retrieval and reading comprehension.

  • Retrieval: This stage first uses a dense retriever (PubMedBERT fine-tuned on COVID-QA) to find relevant documents based on semantic similarity. This is followed by a BM25 re-ranking step to incorporate keyword matching, which is crucial for specialized domains like biomedical science. To further enhance the results, a diversity re-ranking mechanism is applied, clustering the retrieved documents and sampling from each cluster to ensure a broader range of perspectives and information.
  • Reading Comprehension: Once relevant documents are retrieved, a reading comprehension model (a variant of BioBERT fine-tuned on COVID-QA and SQuAD2.0) extracts answer spans from these documents. A novel aspect here is the ability to extract multiple answer spans from a single document, improving coverage and evaluation results. The documents are then re-ranked based on the confidence scores of the extracted answers.

Datasets

  • COVID-QA: Used for fine-tuning both the retrieval and reading comprehension models. It's a COVID-19 QA dataset with question-answer pairs and context documents from PubMed.
  • CORD-19: The primary document corpus used for the open-domain retrieval task. This dataset is continuously updated with new COVID-19 research papers.
  • SQuAD2.0: Used in conjunction with COVID-QA for fine-tuning the BioBERT model for reading comprehension, providing a broader out-of-domain dataset.

Key Insights and Ideas

  • Leveraging Pre-trained Models for Emergent Domains: Fine-tuning general biomedical language models (like PubMedBERT and BioBERT) on a small, in-domain dataset (COVID-QA) is an effective strategy to "warm-start" systems for emergent domains, overcoming data scarcity.
  • Hybrid Retrieval for Robustness: Combining dense retrieval (semantic similarity) with BM25 (keyword matching) provides a more robust retrieval system, capturing both nuanced meaning and specific terminology important in scientific literature.
  • Diversity is Key: Simply retrieving the most relevant documents isn't enough. Applying diversity re-ranking ensures users see a broader spectrum of information, preventing them from getting stuck with redundant or narrowly focused results.
  • Multiple Answer Spans Enhance Coverage: Allowing the system to extract multiple answer spans from a document, rather than just one, significantly improves its ability to capture all relevant information within a text.

Key Results

The paper demonstrates the effectiveness of their hybrid retrieval model, showing improved performance over individual dense retrieval or BM25 methods on the COVID-QA test set, with a fuzzy matching score of 39.16 (FM@50) for the BioBERT model with multiple answer spans. The reading comprehension model also benefits from fine-tuning on both in-domain and out-of-domain data. The authors also provide a functional online demo of their system.

Limitations and Future Work

The authors acknowledge that the reduced data size in emergent domains can lead to lower answer precision. While their system aims to mitigate this, it remains a challenge. They also note that the COVID-QA dataset, while useful, is relatively small. Future work could involve exploring more advanced techniques for handling data scarcity and further refining the diversity and answer extraction components. The system is presented as a model for quickly developing similar systems for other emergent domains, suggesting adaptability as a key future direction.

Summary updated

BibTeX
@inproceedings{levy2021open,
  author = {Sharon Levy and Kevin Mo and Wenhan Xiong and William Yang Wang},
  title = {{Open-Domain Question-Answering for COVID-19 and Other Emergent Domains}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=maU1EPAJ9pr},
}
Figure