COVID-19 Kaggle Literature Organization

Maksim Ekin Eren, Nick Solovyev, Edward Raff, Charles Nicholas, Ben Johnson

TL;DR

Researchers developed an unsupervised machine learning pipeline to organize the explosion of COVID-19 scientific literature. By combining TF-IDF vectorization, PCA, and k-means clustering with t-SNE visualization, they created an interactive map that allows scientists to navigate thousands of papers by topic, helping them find relevant research in a rapidly growing field.

What Problem Does This Paper Address?

In the early months of 2020, the scientific community faced an unprecedented "infodemic." Research on SARS-CoV-2 was being published at a staggering rate—over 23,000 papers by May 2020, with the volume doubling every 20 days.

This massive influx of information left health professionals and researchers overwhelmed. Traditional search methods were insufficient for discovering latent connections between papers or getting a "bird's-eye view" of the research landscape. The authors sought to build a tool that could automatically group similar papers together and visualize these relationships, making it easier to navigate the literature without manual tagging.

Methods

The authors built a multi-stage NLP and machine learning pipeline:

  1. Pre-processing: They cleaned the dataset by removing duplicates, non-English papers, and entries containing only abstracts. They used SciSpacy, a specialized biomedical parser, for tokenization and removed domain-specific "stop words" (like "doi" or "fig").
  2. Vectorization: The cleaned text was converted into numerical feature vectors ($X_1$) using TF-IDF (Term Frequency-Inverse Document Frequency), limited to $2^{12}$ features to manage memory and filter noise.
  3. Dimensionality Reduction for Clustering: To make clustering more efficient, they applied Principal Component Analysis (PCA) to reduce the feature space while retaining 95% of the variance ($X_2$).
  4. Clustering: They used k-means clustering on the PCA-reduced vectors. To find the optimal number of clusters ($k$), they used the "elbow method," identifying $k=20$ as the point where increasing the number of clusters yielded diminishing returns in reducing distortion.
  5. Visualization: For the final 2D map, they used t-SNE (t-distributed Stochastic Neighbor Embedding), a non-linear dimensionality reduction technique specifically designed to preserve local structures in high-dimensional data.

Datasets

The study utilized the CORD-19 (COVID-19 Open Research Dataset). At the time of the study, this dataset (hosted on Kaggle) contained over 141,000 scholarly articles related to COVID-19 and the broader corona family of viruses. After cleaning, the authors focused on a corpus of 49,967 papers with full body text.

Key Insights and Ideas

  • Terminology Drives Clustering: The authors found that "niche" topics, such as animal infection and viral reservoirs, formed very tight clusters. This is likely because these papers use highly specific, consistent terminology (e.g., specific species names) that the TF-IDF algorithm easily identifies.
  • Interdisciplinary Spread: Conversely, broader topics like epidemiology or virology were more "spread out" on the map. These fields share overarching themes and vocabulary with many other clusters, leading to less distinct boundaries.
  • Methodological Agreement: Even though t-SNE (for visualization) and k-means (for labeling) were applied independently, they largely agreed on the bounds of major clusters. This suggests the underlying thematic structure of the literature is robust.
  • Cross-Domain Utility: While built for COVID-19, the pipeline proved adaptable. A biology researcher successfully used the same code to cluster literature on tropical reef fish and plastic pollution, demonstrating the tool's generalizability to other scientific crises.

Key Results

  • Optimal Granularity: The "elbow curve" analysis determined that 20 clusters provided a balanced organization of the CORD-19 dataset.
  • Qualitative Validation: Biomedical engineers from Utah State University used the tool to scan for specific drug effectiveness and broad-spectrum antivirals, confirming that the tool successfully grouped related, high-value research.
  • Community Impact: The project’s Jupyter notebook became a popular resource on Kaggle, being forked or visited an average of ten times a day during the height of the research surge.

Limitations and Future Work

  • The 2D Constraint: The authors acknowledge that projecting high-dimensional data into 2D inevitably loses some structural information. Some clusters that appear to overlap in the 2D plot might be distinct in higher-dimensional space.
  • Cluster Splitting: The choice of $k=20$ is a heuristic. A higher $k$ would reduce mathematical "distortion" but might split meaningful, large clusters into fragments that are harder for a human to interpret.
  • Dynamic Updating: While the paper describes a static snapshot, the authors note the need for tools that can handle the continuous, daily influx of new research.

Summary updated

Abstract

The world has faced the devastating outbreak of Severe Acute Respiratory Syndrome Coronavirus-2 (SARS-CoV-2), or COVID-19, in 2020. Research in the subject matter was fast-tracked to such a point that scientists were struggling to keep up with new findings. With this increase in the scientific literature, there arose a need for organizing those documents. We describe an approach to organize and visualize the scientific literature on or related to COVID-19 using machine learning techniques so that papers on similar topics are grouped together. By doing so, the navigation of topics and related papers is simplified. We implemented this approach using the widely recognized CORD-19 dataset to present a publicly available proof of concept.
BibTeX
@inproceedings{eren2020covid,
  author = {Maksim Ekin Eren and Nick Solovyev and Edward Raff and Charles Nicholas and Ben Johnson},
  title = {{COVID-19 Kaggle Literature Organization}},
  booktitle = {DocEng},
  year = {2020},
  url = {https://openreview.net/forum?id=w6CU0hd0bSo},
}

Loading related content...

Figure