COVID-19: Comparative Analysis of Methods for Identifying Articles Related to Therapeutics and Vaccines without Using Labeled Data

Mihir Parmar, Ashwin Karthik Ambalavanan, Hong Guan, Rishab Banerjee, Jitesh Pabla, Murthy V. DevarakondaarXiv 2021 · OpenReview

Abstract

Here we proposed an approach to analyze text classification methods based on the presence or absence of task-specific terms (and their synonyms) in the text. We applied this approach to study six different transfer-learning and unsupervised methods for screening articles relevant to COVID-19 vaccines and therapeutics. The analysis revealed that while a BERT model trained on search-engine results generally performed well, it miss-classified relevant abstracts that did not contain task-specific terms. We used this insight to create a more effective unsupervised ensemble.

Introduction

The COVID-19 pandemic led to an unprecedented surge in scientific publications, creating a significant challenge for researchers attempting to navigate the COVID-19 Open Research Dataset (CORD-19). Identifying specific subsets of literature, such as those concerning vaccines or therapeutics, is critical but labor-intensive. While supervised machine learning offers a solution, the requirement for large-scale labeled training data is often prohibitive in rapidly evolving public health crises. This study evaluates unsupervised and transfer-learning methods to automate this screening process and introduces a novel error-analysis framework to understand model performance beyond aggregate metrics.

Methods

The researchers addressed the task as a three-class classification problem (vaccine-related, therapeutics-related, or other) using article titles, abstracts, and journal names. Six distinct unsupervised or transfer-learning approaches were evaluated:

  1. Next Sentence Prediction (NSP): Utilizing SciBERT to determine if article text logically follows handcrafted prototype passages.
  2. Clinical Hedges (CH): SciBERT fine-tuned on the MEDLINE Clinical Hedges dataset to identify treatment-related content.
  3. Semantic Textual Similarity (STS): Measuring the distance between articles and query sentences using a clinical-similarity BERT model.
  4. Lexicon-based Similarity Scoring (LSS): Calculating cosine similarity between BioBERT embeddings and specific seed words.
  5. Search-results trained approach (GS): A SciBERT model weakly supervised on Google Scholar results for specific keyword queries.
  6. Micro-scorers (MS): A TF-IDF based system comparing article vectors against representative query vectors.

To improve performance, the authors developed a Majority Voting (MV) ensemble strategy, specifically focusing on a targeted three-model ensemble (MV_3) comprising the most complementary systems (CH, GS, and MS).

Evaluation Framework: The 2x2 Error Analysis Matrix

A key contribution of this work is the 2x2 error analysis matrix. Instead of relying solely on standard performance metrics, the authors categorized the test set based on two dimensions: the actual relevance of the article (manual judgment) and the explicit presence of task-specific keywords (e.g., "vaccine").

2x2 Error Analysis Matrix defining Categories 1-4 based on manual judgment and keyword presence.

This framework revealed that models like the Google Scholar-trained (GS) approach, while precise, often failed on "Category 3" articles—papers that are relevant but lack explicit keywords. Conversely, models like MS and CH were more effective at identifying these implicit references, highlighting the complementary nature of different unsupervised techniques.

Results

Evaluation on a manually annotated test set of 203 CORD-19 articles showed that individual models reached a performance ceiling. Both the GS and MS methods achieved the highest individual F-measures of 0.50. However, their strengths differed: GS provided high precision, while MS offered superior recall.

Performance metrics (Precision, Recall, F-measure) for individual systems and ensembles.

The study demonstrated that a "kitchen sink" ensemble of all six models (MV_6) actually performed worse than the best individual models (F-measure 0.49). In contrast, the strategic MV_3 ensemble—designed to cover the blind spots identified in the 2x2 analysis—achieved a significant performance boost, reaching an F-measure of 0.65 with a precision of 0.59 and recall of 0.69.

Conclusion

This research highlights the limitations of relying on keyword-based relevance for scientific literature screening. By employing a structured error-analysis framework, the authors were able to identify complementary failure modes in unsupervised models and construct a highly effective ensemble. While the small size of the manual test set and the need for domain-specific queries are limitations, the 2x2 categorization approach provides a robust template for improving text classification tasks where labeled data is scarce.

Summary updated

Abstract

Here we proposed an approach to analyze text classification methods based on the presence or absence of task-specific terms (and their synonyms) in the text. We applied this approach to study six different transfer-learning and unsupervised methods for screening articles relevant to COVID-19 vaccines and therapeutics. The analysis revealed that while a BERT model trained on search-engine results generally performed well, it miss-classified relevant abstracts that did not contain task-specific terms. We used this insight to create a more effective unsupervised ensemble.
BibTeX
@misc{parmar2021covid,
  author = {Mihir Parmar and Ashwin Karthik Ambalavanan and Hong Guan and Rishab Banerjee and Jitesh Pabla and Murthy V. Devarakonda},
  title = {{COVID-19: Comparative Analysis of Methods for Identifying Articles Related to Therapeutics and Vaccines without Using Labeled Data}},
  year = {2021},
  eprint = {2101.02017},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.02017},
}
Figure