COVID-19 information retrieval with deep-learning based semantic search, question answering, and abstractive summarization

Andre Esteva, Anuprit Kale, Romain Paulus, Kazuma Hashimoto, Wenpeng Yin, Dragomir Radev, Richard SocherNpj Digit. Medicine 2021 · DOI

Introduction

The COVID-19 pandemic led to an unprecedented surge in scientific literature, often referred to as a "paper-demic." By late 2020, the corpus exceeded 400,000 articles, with thousands more added weekly. For healthcare professionals and policy makers, finding specific, actionable answers within this volume using traditional keyword search is increasingly difficult. CO-Search was developed as a multi-stage semantic search engine to bridge the gap between simple keyword matching and deep semantic understanding.

The CO-Search Architecture

CO-Search employs a cascaded architecture designed to balance speed and precision. The system operates in two primary stages: a hybrid retriever and a sophisticated re-ranker.

Hybrid Retrieval and Self-Supervised Training

The first stage filters the massive corpus down to the top 1,000 most relevant documents. It uses a hybrid approach, combining Siamese-BERT (SBERT) for semantic encoding with BM25 and TF-IDF for lexical matching. This ensures that the system captures both the conceptual meaning of a query and specific technical terminology.

A significant challenge in developing such models for niche domains is the lack of labeled training data (query-document pairs). CO-Search addresses this through a novel text augmentation technique. By treating citations within a paper as "queries" and the surrounding paragraphs as "relevant documents," the authors generated millions of training tuples without manual human labeling.

CO-Search training pipeline using citation-paragraph pairs to train the Siamese-BERT network.

Semantic Understanding and Clustering

The effectiveness of this training is evident in the model's ability to cluster documents by nuanced sub-topics. Unlike traditional search engines that might conflate different aspects of the virus, CO-Search can distinguish between distinct research areas such as animal models, testing protocols, and social impacts.

t-SNE visualization demonstrating the model's ability to semantically cluster diverse COVID-19 research topics.

Re-ranking for Utility

After the initial retrieval, the top 1,000 documents undergo a more intensive re-ranking process. This stage moves beyond simple similarity to evaluate utility through two modules:

  1. Question Answering (QA): This module extracts potential answer spans and scores documents based on their ability to directly answer the user's query.
  2. Abstractive Summarization (AS): Using a modified GPT-2 model, the system generates a concise summary of the document and ranks it based on how well the summary aligns with the original query.

Evaluation and Results

CO-Search was benchmarked using the TREC-COVID challenge, a gold-standard evaluation for information retrieval in the context of the pandemic.

  • Top-Tier Performance: CO-Search emerged as the #1 system in the "Automatic" category (requiring no human intervention) across approximately half of the evaluated metrics.
  • Robustness: The system scored highly in Bpref, indicating its reliability even when human-judged relevance labels are sparse, as is common in rapidly evolving datasets.
  • Ablation Insights: Studies showed that the hybrid approach significantly outperformed any single model (SBERT, BM25, or TF-IDF) alone. Furthermore, the re-ranking stage (QA + AS) provided a consistent 1–2% performance boost, demonstrating the value of "reading" for answers.

Conclusion

CO-Search demonstrates that combining hybrid retrieval with functional re-ranking can significantly improve the accessibility of scientific literature during a crisis. While computational latency remains a challenge for real-time deployment, the use of citations as proxy labels provides a powerful blueprint for building search engines in rapidly evolving scientific fields. Future work will explore synthetic query generation and the integration of domain-specific models like SciBERT to further enhance retrieval precision.

Summary updated

BibTeX
@article{esteva2021covid,
  author = {Andre Esteva and Anuprit Kale and Romain Paulus and Kazuma Hashimoto and Wenpeng Yin and Dragomir Radev and Richard Socher},
  title = {{COVID-19 information retrieval with deep-learning based semantic search, question answering, and abstractive summarization}},
  journal = {Npj Digit. Medicine},
  year = {2021},
  doi = {10.1038/s41746-021-00437-0},
}
Figure