CAiRE-COVID: A Question Answering and Query-focused Multi-Document Summarization System for COVID-19 Scholarly Information Management

Dan Su, Yan Xu, Tiezheng Yu, Farhad Bin Siddique, Elham J. Barezi, Pascale Fung

TL;DR

The COVID-19 pandemic generated an overwhelming volume of scientific literature, making it difficult for researchers to stay updated. CAiRE-COVID addresses this by providing a system that answers specific questions and generates concise, query-focused summaries from this vast corpus, significantly improving information retrieval and management for medical professionals.

What Problem Does This Paper Address?

The rapid and unprecedented surge in scientific publications related to COVID-19 presented a significant challenge for the medical research community and the public. Sifting through thousands of articles to find answers to critical questions about transmission, risk factors, genetics, and more, was a time-consuming and inefficient process. Existing information retrieval systems often lacked the ability to synthesize information from multiple documents in a targeted manner, leaving a gap in managing and digesting this rapidly expanding body of knowledge.

Methods

CAiRE-COVID is built upon three core modules:

  1. Document Retriever: This module first identifies the most relevant documents (articles or paragraphs) from the CORD-19 dataset based on a user's query. It employs a search engine (Anserini, built on Lucene) that indexes paragraphs from the full text, titles, and abstracts.
  2. Relevant Snippet Selector: This module pinpoints the most relevant answer snippets within the retrieved documents. It utilizes a neural Question Answering (QA) system, specifically an ensemble of two models: the HLTC-MRQA model (trained on multiple QA datasets for generalization) and the BioBERT QA model (fine-tuned on biomedical data). This ensemble approach enhances both generalization and domain-specific accuracy. The system then re-ranks paragraphs based on a combined score of confidence from the QA models and keyword matching.
  3. Query-focused Multi-Document Summarizer: This module generates both abstractive and extractive summaries tailored to the user's query. For abstractive summarization, it fine-tunes the BART model, incorporating answer relevance from the QA module by concatenating predicted answer spans and the query to the input paragraphs. For extractive summarization, it identifies sentences containing answer spans and re-ranks them based on answer relevance to form the summary.

Datasets

  • CORD-19: The primary dataset used for information retrieval, consisting of over 158,000 scholarly articles about COVID-19 and related coronaviruses.
  • CovidQA: A dataset recently released for zero-shot evaluation of QA models on COVID-19 related questions.
  • DUC Datasets (DUC 2005, DUC 2006, DUC 2007): Standard datasets for evaluating Query-Focused Summarization (QFS) tasks.
  • Debatepedia: A large-scale QFS dataset used for evaluating summarization performance, particularly for abstractive summarization.

Key Insights and Ideas

  • Ensembling QA models is crucial: Combining a generalist QA model (HLTC-MRQA) with a domain-specific one (BioBERT) significantly improves performance and robustness, especially in a new domain like COVID-19.
  • Answer relevance is key for summarization: Simply using retrieved paragraphs isn't enough. Incorporating the QA model's confidence and predicted answer spans into the summarization process leads to more accurate and relevant summaries.
  • Query paraphrasing can help, but needs careful implementation: While breaking down complex queries can be beneficial, the authors found that their initial automatic paraphrasing methods didn't yield satisfactory improvements and were excluded from the deployed system.
  • The system is adaptable: The modular design and the use of pre-trained language models allow for potential generalization to other scientific domains or future pandemics.

Key Results

  • CAiRE-COVID won one of the tasks in the Kaggle COVID-19 Open Research Dataset Challenge, validated by medical experts.
  • The QA module showed competitive performance on the CovidQA dataset, outperforming the T5 model on recall and marginally on precision for keyword questions.
  • The query-focused multi-document summarizer achieved consistent ROUGE score improvements over baseline methods on both DUC and Debatepedia datasets by incorporating answer relevance.
  • The system has been launched as a website (CAIRE-COVID) for real-time interaction and its code has been open-sourced.

Limitations and Future Work

  • Redundancy in abstractive summaries: The authors acknowledge that their current abstractive summarization method might include some redundancy, which they consider acceptable for the current stage.
  • Automatic query paraphrasing: The effectiveness of automatic query paraphrasing needs further exploration to ensure it provides a performance boost.
  • Future work: The authors suggest exploring more automatic methods for query paraphrasing and further improving the system's generalization capabilities for broader applications.

Summary updated

BibTeX
@inproceedings{su2020caire,
  author = {Dan Su and Yan Xu and Tiezheng Yu and Farhad Bin Siddique and Elham J. Barezi and Pascale Fung},
  title = {{CAiRE-COVID: A Question Answering and Query-focused Multi-Document Summarization System for COVID-19 Scholarly Information Management}},
  booktitle = {NLP4COVID @ EMNLP},
  year = {2020},
  doi = {10.18653/v1/2020.nlpcovid19-2.14},
}
Figure