Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learned

Edwin Zhang, Nikhil Gupta, Rodrigo Nogueira, Kyunghyun Cho, Jimmy Lin

Abstract

We present the Neural Covidex, a search engine that exploits the latest neural ranking architectures to provide information access to the COVID-19 Open Research Dataset (CORD-19) curated by the Allen Institute for AI. This web application exists as part of a suite of tools that we have developed over the past few weeks to help domain experts tackle the ongoing global pandemic. We hope that improved information access capabilities to the scientific literature can inform evidence-based decision making and insight generation. This paper describes our initial efforts and offers a few thoughts about lessons we have learned along the way.

Introduction

When the COVID-19 Open Research Dataset (CORD-19) was released in March 2020, it quickly grew to include tens of thousands of scholarly articles. For clinicians, virologists, and public health officials, the sheer volume of information made it difficult to find specific, evidence-based answers to urgent questions. While standard keyword search engines are widely available, they often struggle with the complex, nuanced queries typical of scientific research. Furthermore, state-of-the-art neural search models, while highly effective in academic benchmarks, are rarely deployed in production-ready environments during a crisis. The Neural Covidex was developed to bridge this gap, providing a functional, end-to-end neural search tool in just two weeks.

System Architecture and Methods

The Neural Covidex employs a multi-stage ranking architecture designed to balance retrieval efficiency with the deep semantic understanding of transformer models.

Initial Retrieval

The first stage uses the Anserini IR toolkit (via Pyserini) to perform keyword-based retrieval using BM25 scoring. To address the varying lengths of scientific papers—which can range from short abstracts to lengthy reviews—the team implemented a paragraph-level index. Each entry in the index consists of the article's title, abstract, and a single paragraph. This approach prevents the system from being biased toward longer documents and ensures that the subsequent neural stages receive manageable chunks of text.

The basic Covidex interface showing keyword-based search results and filtering options.

Neural Reranking

The top candidates from the initial retrieval are passed to a T5-base sequence-to-sequence model for reranking. The model is presented with the query and the document text and is fine-tuned to output "true" or "false" tokens to indicate relevance. To handle T5's 512-token input limit, the system applies a sliding window of 10 sentences with a stride of 5 across the text, using the highest probability span to determine the final document score.

Unsupervised Highlighting

To improve the user experience, the system incorporates an unsupervised highlighting feature. Using a pretrained BioBERT model, the system calculates the cosine similarity between the hidden vectors of the query and the retrieved sentences. The most relevant sentences are then highlighted in the user interface to help researchers quickly scan for pertinent information.

The Neural Covidex interface featuring natural language query support and unsupervised highlighting of relevant text spans.

Key Insights and Lessons Learned

Several practical lessons emerged from the rapid deployment of this system:

  1. Zero-Shot Transfer is Effective: Despite having no specific training data for CORD-19, the T5 model—fine-tuned on the general-domain MS MARCO passage dataset—showed remarkable effectiveness in the biomedical domain. This suggests that modern neural rankers are highly robust across different corpora.
  2. UI/UX is Paramount: Technical ranking performance is only one part of the equation. Users expressed a strong preference for features like unsupervised highlighting, which directly supported their task of information extraction. A slightly less accurate ranker with a superior interface is often more useful in practice than a "perfect" model with poor presentation.
  3. The Research-to-Production Gap: Moving from offline research code to a live system required addressing non-trivial engineering challenges, such as query-time GPU latency (optimized to ~2 seconds) and fixing minor tokenization artifacts (e.g., ensuring "COVID-19" rendered correctly) that could otherwise erode user trust.

Limitations and Future Work

The primary limitation of this initial deployment is the lack of formal quantitative evaluation. Because no relevance judgments existed for CORD-19 at the time of launch, the system's performance remains anecdotal. Future work will involve participating in community evaluation efforts (like TREC-COVID) and refactoring the current monolithic architecture into scalable microservices. Additionally, the team plans to explore alternative paradigms, such as daily email digests or notification systems, to better align with the workflows of frontline medical professionals.

Summary updated

Abstract

We present the Neural Covidex, a search engine that exploits the latest neural ranking architectures to provide information access to the COVID-19 Open Research Dataset curated by the Allen Institute for AI. This web application exists as part of a suite of tools that we have developed over the past few weeks to help domain experts tackle the ongoing global pandemic. We hope that improved information access capabilities to the scientific literature can inform evidence-based decision making and insight generation. This paper describes our initial efforts and offers a few thoughts about lessons we have learned along the way.
BibTeX
@inproceedings{zhang2020rapidly,
  author = {Edwin Zhang and Nikhil Gupta and Rodrigo Nogueira and Kyunghyun Cho and Jimmy Lin},
  title = {{Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learned}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure