BERT-Based Natural Language Processing of Drug Labeling Documents: A Case Study for Classifying Drug-Induced Liver Injury Risk

Yue Wu, Zhichao Liu, Leihong Wu, Minjun Chen, Weida TongFrontiers Artif. Intell 2021 · DOI

TL;DR

This paper introduces a BERT-based natural language processing (NLP) approach to automatically classify drug labeling documents for drug-induced liver injury (DILI) risk. The developed models demonstrate high accuracy and portability across regulatory agencies, offering a promising way to modernize and accelerate regulatory science.

What Problem Does This Paper Address?

The U.S. Food and Drug Administration (FDA) is responsible for regulating a vast array of consumer products, including medications. This involves processing an enormous volume of documents, a task that is time-consuming and labor-intensive when done manually. A critical aspect of this is identifying potential safety risks, such as drug-induced liver injury (DILI), which are described in drug labeling documents. Traditional methods for identifying DILI risk rely on keyword matching, which can be imprecise due to the complex language and lack of standardized terminology in drug labels. This paper addresses the need for a more efficient and accurate method to classify DILI risk in drug labeling documents, aiming to support regulatory science and improve drug safety surveillance.

Methods

The researchers developed and compared three models for DILI risk classification:

  • Deep Learning-based Model (BERT-based): This model utilizes BERT (Bidirectional Encoder Representations from Transformers), a state-of-the-art language model. It incorporates a BERT-based Named Entity Recognition (NER) module for context classification (identifying liver-related sentences) and a BERT-based sentence classification module to determine if a sentence suggests DILI risk.
  • Hybrid Deep Learning-based Model: This model also uses a BERT-based sentence classification module but employs a simpler, string pattern-matching approach for context classification.
  • Keywords-based Model: This is a conventional approach that relies on pre-defined DILI keywords to identify risk.

The core innovation lies in leveraging BERT's ability to understand the semantic meaning of sentences, moving beyond simple keyword matching. The models were trained and evaluated using a hierarchical approach, first classifying sentences and then aggregating these classifications to determine the document's DILI risk.

Datasets

  • FDA Drug Labeling Documents: These were retrieved from DailyMed and formed the primary dataset for training and internal validation.
  • EMA Drug Labeling Documents: Documents for drugs marketed in the UK (representing EMA-approved drugs) were used for external validation, assessing the model's cross-agency portability.
  • DILIrank Dataset: This dataset provided DILI risk annotations for over 1,000 drugs, used to identify representative drugs for the study.

The study focused on the "Warnings and Precautions" section of FDA labels and the "Special warnings and precautions for use" section of EMA labels, as these sections contain the most complex descriptions of adverse drug reactions.

Key Insights and Ideas

  • BERT's Semantic Understanding is Key: The study highlights that BERT-based models can capture the nuanced semantic meanings within drug labeling text, which is crucial for accurately identifying DILI risk beyond simple keyword presence.
  • Portability Across Agencies: A significant finding is that models trained on FDA data performed comparably when validated on EMA data, demonstrating the robustness and generalizability of the approach across different regulatory bodies.
  • Hierarchical Classification Works: The sentence-level classification followed by document-level aggregation proved effective, allowing for interpretable identification of sentences contributing to the DILI risk assessment.
  • NER Enhances Contextualization: The BERT-based NER module for context classification was shown to be more effective than simple string filtering, improving the selection of relevant sentences for DILI analysis.
  • Higher Recall is Preferred for Safety: The study emphasizes that for DILI detection, a higher recall (minimizing false negatives) is more critical than precision, as false negatives represent missed safety signals.

Key Results

  • The Deep Learning-based Model achieved a recall of 1.00 and a precision of 0.78 on FDA test documents, and a recall of 1.00 with a precision of 0.71 on EMA validation documents.
  • The Hybrid Deep Learning-based Model also showed strong performance, with a recall of 1.00 and precision of 0.82 on FDA data, and recall of 1.00 with precision of 0.77 on EMA data.
  • Both deep learning models significantly outperformed the Keywords-based Model, which had lower MCC scores and higher false positive rates, particularly for contraindications and hypersensitivity reactions.
  • The BERT-based NER model demonstrated excellent performance in identifying liver-related sentences, with an F1 score of 0.98 at the token level and a recall of 0.99 at the sentence level.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the dataset size, particularly for document-level classification, is relatively small due to the rarity of DILI as an adverse event.
  • Single Topic Focus: The current framework was evaluated for DILI risk only. Future research should explore its extensibility to other safety topics (e.g., cardiotoxicity, drug indications, drug-drug interactions).
  • In-Domain Training: While BERT models are powerful, further improvements could be achieved through in-domain training on drug labeling text.
  • Exploring Other Models: The study focused on BERT, but other transformer models (e.g., GPT-2, XLNet) could also be investigated.
  • Interpretability: While the hierarchical approach aids interpretability, further work could explore more advanced methods for understanding model predictions.
  • Extensibility: The authors suggest that the framework for dataset creation and context classification can be extended to other topics, and the outputs could be used for information retrieval pipelines.

Summary updated

BibTeX
@article{wu2021bert,
  author = {Yue Wu and Zhichao Liu and Leihong Wu and Minjun Chen and Weida Tong},
  title = {{BERT-Based Natural Language Processing of Drug Labeling Documents: A Case Study for Classifying Drug-Induced Liver Injury Risk}},
  journal = {Frontiers Artif. Intell.},
  year = {2021},
  doi = {10.3389/frai.2021.729834},
}
Figure