TL;DR
The COVID-19 pandemic generated an overwhelming amount of information, making it difficult for various stakeholders (from clinicians to the general public) to find timely and relevant answers. This paper introduces EPIC-QA, a novel dataset designed to train and evaluate automatic question-answering systems that can cater to different levels of expertise, recognizing that a "one-size-fits-all" answer is often insufficient.
What Problem Does This Paper Address?
The sheer volume and rapid evolution of information during the COVID-19 pandemic highlighted a critical challenge: how to efficiently deliver accurate and understandable answers to a diverse range of users. Traditional search engines often return too many irrelevant results, and information seekers struggle to find answers tailored to their specific needs and background knowledge. This is particularly true when dealing with complex scientific topics where experts require detailed, nuanced information, while the general public needs clear, accessible explanations. The paper addresses the need for question-answering systems that can bridge this gap by providing context-aware answers suitable for different audiences.
Methods
The authors developed the EPIC-QA dataset with two primary research goals in mind: answer extraction and end-to-end question answering.
- Answer Extraction: This involves identifying relevant passages or answers within pre-retrieved documents. The questions for this part of the dataset were adapted from TREC-COVID, and the dataset includes document-level relevance judgments.
- End-to-End Question Answering: This approach trains models to jointly identify relevant passages and extract answers. For this, a novel set of questions was created, and the dataset provides relevant documents and answer annotations without pre-computed retrieval rankings.
A key methodological innovation is the creation of a dataset that explicitly considers the different information needs of experts and the general public. This is achieved by:
- Generating Dual-Level Questions: Questions were collected from both expert sources (e.g., researchers, clinicians) and the general public, with an emphasis on overlapping topics to facilitate comparison.
- Curating Diverse Document Sources: The dataset includes scientific research articles, government websites, news articles, and social media posts to cover a wide spectrum of information.
- Expert Annotation: Human assessors with extensive experience (medical indexers) meticulously annotated the data, creating "nuggets" (atomic facts) that answer questions and then marking which sentences in the documents support these nuggets. This rigorous annotation process ensures the quality and depth of the dataset.
Datasets
The core contribution of this paper is the introduction of the Epidemic Question Answering (EPIC-QA) dataset. This dataset is divided into two main parts:
- Answer Extraction Collection:
- 45 expert-level questions and 42 general public questions.
- Associated with document-level relevance judgments from TREC-COVID.
- End-to-End Question Answering Collection:
- 30 expert-level questions and 30 general public questions.
- Includes relevant documents and answer annotations, but no pre-computed retrieval rankings.
The dataset comprises:
- Questions: Manually produced questions from both expert and general public sources.
- Documents: A collection of biomedical research articles (adapted from CORD-19) and public-facing online documents (from sources like CDC, WHO, MedlinePlus). These documents are pre-processed with explicit context and sentence boundaries.
- Answers: Manually produced answers, including relevant document excerpts, with sentence-level annotations indicating which answers are evidenced by each sentence.
Key Insights and Ideas
- The "One-Size-Fits-All" Answer is a Myth: The paper strongly suggests that satisfactory answers to the same question can differ significantly based on the user's expertise level. An expert might need detailed mechanisms, while a layperson needs a high-level overview.
- Diverse Information Sources are Crucial: Effectively answering questions requires drawing from a broad range of sources, from peer-reviewed research to public health advisories and even social media, to cater to different information needs and accessibility levels.
- Annotation is Key for Nuance: The detailed, expert-driven annotation process for EPIC-QA is vital. It goes beyond simple fact extraction to identify "nuggets" of information and link them to specific sentences, enabling a deeper understanding of answer provenance.
- End-to-End Systems Show Promise: The technical validation suggests that end-to-end question-answering systems, despite being more challenging, are performing well on this dataset, indicating their potential for real-world application.
Key Results
The technical validation involved a community evaluation at the 2020 Text Analysis Conference (TAC).
- Answer Extraction: Teams submitted results for 17 expert and 10 general public questions. The paper reports performance using a modified Normalized Discounted Cumulative Gain (NDCG) metric called Normalized Discount Novelty Score (NDNS), with various scoring variations (Exact, Partial, Relaxed). Top-performing systems achieved NDNS scores in the range of 0.3-0.48 for expert questions and 0.25-0.48 for general public questions, depending on the scoring metric.
- End-to-End: 16 submissions for expert questions and 12 for general public questions were evaluated. Teams generally performed better on end-to-end tasks, with top scores reaching around 0.42 for expert questions and 0.36 for general public questions.
- Qualitative Findings: The evaluation demonstrated that systems could indeed discover useful answers and that the expected answers varied substantially between expert and public stakeholders. The paper highlights that systems need to account for these varying levels of understanding.
Limitations and Future Work
- Dataset Size: While the annotation is thorough, the number of questions in the dataset (e.g., 87 for answer extraction, 60 for end-to-end) is relatively small. The authors acknowledge this, noting that each question is associated with many passages and sentence-level annotations.
- Answer Accuracy vs. Evidence: The paper states that "any statement that answers the question is considered an 'answer' regardless of whether or not the answer is factually accurate." This approach aims to capture the landscape of information available, including potentially outdated or incorrect answers, which is valuable for understanding information retrieval but might require further filtering for downstream applications focused solely on factual accuracy.
- Generalizability: While the dataset is designed to be generalizable to other domains (legal, finance), its primary focus is on epidemic-related questions. Future work could involve expanding the dataset to cover a wider range of domains and question types.
- System Performance: While systems performed reasonably well, there is still room for improvement in accurately capturing the nuances of expert versus public-level answers and in handling the dynamic nature of information during a crisis. The authors hope the dataset will stimulate further research in these areas.
Summary updated