CAiRE-COVID: A Question Answering and Query-focused Multi-Document Summarization System for COVID-19 Scholarly Information Management

Dan Su, Yan Xu, Tiezheng Yu, Farhad Bin Siddique, Elham J. Barezi, Pascale Fung

Introduction

The COVID-19 pandemic triggered an unprecedented surge in scientific literature, with the CORD-19 dataset growing to over 158,000 articles in its early stages. For medical researchers, the challenge shifted from finding individual papers to synthesizing information across a vast corpus to answer complex questions regarding risk factors, genetics, and transmission. Standard search engines often fail to provide the necessary synthesis, returning noisy results that lack specific context. CAiRE-COVID was developed as an end-to-end system to address this by providing evidence-backed, query-focused summaries.

System Architecture

The CAiRE-COVID pipeline consists of three primary stages designed to transform a natural language query into a structured summary.

The CAiRE-COVID system architecture, illustrating the three-stage pipeline from user query to multi-document summary.

1. Document Retriever

The system utilizes the Anserini toolkit (built on Lucene) to index paragraphs from the CORD-19 dataset. To handle the complexity of medical queries, a query paraphrasing module breaks down long questions into simpler sub-queries, improving retrieval precision.

2. Relevant Snippet Selector (QA Engine)

Retrieved paragraphs are processed by an ensemble of two neural models to identify and highlight specific "answer spans":

  • HLTC-MRQA: An XLNet-based model trained on diverse QA datasets to ensure cross-domain generalization.
  • BioBERT: A BERT model pre-trained on biomedical literature (PubMed) to provide domain-specific expertise.

3. Query-focused Multi-Document Summarizer

The system provides both extractive and abstractive summaries. The extractive component uses ALBERT embeddings to select representative sentences. The abstractive component employs a novel "answer-aware" BART model. Unlike standard summarizers, this model receives the original query and the predicted answer spans concatenated with the context, forcing the model to focus on the most relevant evidence.

Experimental Setup and Datasets

The system was evaluated using the CORD-19 corpus and the CovidQA dataset (124 question-article pairs). Due to the lack of large-scale COVID-specific summarization benchmarks at the time, the authors utilized gold-standard datasets such as DUC (2005-2007) and Debatepedia to validate the summarization module. The QA engine was pre-trained on multi-task datasets including SQuAD, NewsQA, and TriviaQA.

Results and Discussion

The "answer-aware" approach proved to be a significant innovation. By explicitly providing the model with answer spans, the quality of generated summaries improved across all metrics.

Performance comparison of different BART configurations on the Debatepedia dataset, highlighting the superiority of the answer-aware (C,A,Q) approach.

As shown in the evaluation on Debatepedia, the BART(C,A,Q) configuration—incorporating Context, Answer, and Query—achieved a ROUGE-L F1 score of 12.18, a notable improvement over the baseline BART(C) score of 10.70. Furthermore, the ensemble QA model demonstrated superior robustness, outperforming the T5 baseline in recall (R@3) on the CovidQA dataset. The system's real-world utility was recognized when it won a task in the Kaggle CORD-19 Open Research Dataset Challenge, as judged by medical experts.

Conclusion and Future Work

CAiRE-COVID demonstrates the power of combining neural QA with answer-aware summarization for managing specialized information. While successful, the system faces challenges with redundancy in multi-document outputs and the difficulty of fully automating high-quality query paraphrasing. Future efforts will focus on cross-document de-duplication and the development of domain-specific evaluation benchmarks for COVID-19 summarization.

Summary updated

BibTeX
@inproceedings{su2020caire,
  author = {Dan Su and Yan Xu and Tiezheng Yu and Farhad Bin Siddique and Elham J. Barezi and Pascale Fung},
  title = {{CAiRE-COVID: A Question Answering and Query-focused Multi-Document Summarization System for COVID-19 Scholarly Information Management}},
  booktitle = {NLP4COVID @ EMNLP},
  year = {2020},
  doi = {10.18653/v1/2020.nlpcovid19-2.14},
}
Figure