Abstract
Thousands of scientific publications discuss evidence on the efficacy of non-cancer generic drugs being tested for cancer. However, trying to manually identify and extract such evidence is intractable at scale. We introduce a natural language processing pipeline to automate the identification of relevant studies and facilitate the extraction of therapeutic associations between generic drugs and cancers from PubMed abstracts. We annotate datasets of drug-cancer evidence and use them to train models to identify and characterize such evidence at scale. To make this evidence readily consumable, we incorporate the results of the models in a web application that allows users to browse documents and their extracted evidence. Users can provide feedback on the quality of the evidence extracted by our models. This feedback is used to improve our datasets and the corresponding models in a continuous integration system. We describe the natural language processing pipeline in our application and the steps required to deploy services based on the machine learning models.
Introduction
Cancer treatment costs are a global burden, exceeding $1 trillion annually. While the pharmaceutical industry often prioritizes high-cost new drug development, a wealth of potential lies in existing generic drugs. "Drug repurposing"—finding new uses for drugs already approved for other conditions—offers a cost-effective alternative. A classic example is Thalidomide, originally for morning sickness, now a staple in multiple myeloma treatment. However, the evidence for these "hidden gems" is scattered across millions of scientific abstracts, making manual identification an intractable task for researchers.
The NLP Pipeline for Evidence Extraction
To address this, researchers from IBM and Reboot Rx developed an automated Natural Language Processing (NLP) pipeline. This system is designed to identify, extract, and characterize drug-cancer associations from PubMed abstracts at scale, transforming dense scientific text into actionable data.
The pipeline consists of several key stages:
- Targeted Querying: Utilizing the Cochrane highly sensitive search (CHSS) strategy to retrieve relevant documents from PubMed.
- Relevance Filtering: A BERT-based model filters out papers that discuss biological mechanisms without reporting clinical or phenotypic outcomes (like tumor reduction).
- Named Entity Recognition (NER): Using ScispaCy to identify and link drug names and cancer types to standardized databases.
- Classification: The system categorizes the study type (e.g., clinical trial vs. pre-clinical) and the therapeutic association (Effective vs. Non-effective) using RoBERTa and Random Forest models.
Data and Training Insights
The models were trained on specialized datasets, including expert-annotated evidence and data from the Repurposing Drugs in Oncology (ReDO) project. A critical insight during development was the importance of a "phenotypic" filter; by focusing on actual results rather than just the co-occurrence of drug and cancer names, the researchers significantly reduced noise. Additionally, due to the scarcity of published negative results, the therapeutic association model was simplified to a binary "Effective" vs. "Non-effective" classification to maintain high accuracy.
Results and Performance
The pipeline demonstrates robust performance across its various components:
- Filtering Accuracy: 90–95% for identifying relevant abstracts.
- NER Recall: 90% for drug and cancer identification.
- Study Type Classification: 95% accuracy in distinguishing between clinical trials and lab studies.
- Therapeutic Association: 75–83% accuracy in determining efficacy.
Human-in-the-Loop Integration
A unique feature of this system is its integration into a web-based application. This platform allows oncology researchers to browse extracted evidence and, crucially, provide feedback on the model's output.
This feedback loop creates a continuous integration system where the models are regularly retrained and improved based on expert input, ensuring the system remains accurate and up-to-date as new research emerges.
Conclusion and Future Work
While the current system provides a powerful tool for navigating the vast sea of oncology research, it faces challenges such as publication bias toward positive results. Future developments aim to expand the pipeline's scope from abstracts to full-text PDF articles, where more granular data is often located. Ultimately, this NLP-driven approach serves as a bridge, identifying promising generic candidates that can then be prioritized for definitive clinical validation.
Summary updated