Evaluating the Robustness of Biomedical Concept Normalization

Sinchani Chakraborty, Harsh Raj, Srishti Gureja, Tanmay Jain, Atif Hassan, Sayantan BasuTl4nlp 2022

TL;DR

This paper investigates the robustness of biomedical concept normalization models, specifically BERT-based ones, against various input transformations and adversarial attacks. They find that these models are surprisingly brittle, often failing to detect even simple input modifications, and propose new adversarial attack strategies to highlight this vulnerability.

What Problem Does This Paper Address?

Biomedical concept normalization is a crucial task in text mining, aiming to link mentions of medical concepts in text (like "breast cancer") to standardized entries in knowledge bases (like a unique MeSH ID for "breast neoplasm"). This process is vital for accurate information retrieval and decision-making in healthcare. While pre-trained language models like BERT have achieved impressive performance, their susceptibility to adversarial attacks and insensitivity to input variations, as seen in other NLP tasks, remains largely unexplored for biomedical concept normalization. This paper addresses this gap by systematically evaluating how robust these models are to various input perturbations and adversarial attacks, aiming to understand their limitations and explore potential mitigation strategies.

Methods

The authors employ two main categories of methods to test the robustness of BERT-based normalization models:

  • Input Transformations: These are designed to mimic real-world variations in text. They include:
  • Word-level modifications: Hyphenation/dehyphenation, number replacement (e.g., "c9" to "cnine"), disorder synonym/mention term concatenation, and stemming.
  • Word-order variations: Subject-Object Conversion (rearranging parts of a phrase), Copy-Sort, Sort, Reverse, and Shuffle.
  • Gradient-based transformations: Drop, Repeat, Replace, and CopyOne, which aim to alter the most important tokens in a mention.
  • These transformations are applied individually and in chained combinations.
  • Adversarial Attacks: These are designed to create imperceptible changes that fool the model:
  • Adversarial Ranking Attack (Adv-Rank): This attack aims to alter the ranking of candidate concepts for a given mention by adding small, universal perturbations. They explore both mention attacks (MA) and candidate attacks (CA), with MA+ and CA+ indicating raising the rank of the chosen candidate.
  • Least Similar Entity Concatenation (LSEC): This method generates "natural adversarial examples" by concatenating a candidate concept with its most dissimilar sibling from the same ontology parent class. This aims to create plausible but incorrect concept mappings.

Datasets

The experiments were conducted on three standard biomedical normalization datasets:

  • NCBI disease corpus: Contains manually annotated disorder mentions from PubMed abstracts, mapped to MEDIC lexicon concepts.
  • BC5CDR-Disease: Part of the BioCreative V challenge, focusing on disease mentions.
  • BC5CDR-Chemical: Also from BioCreative V, focusing on chemical mentions.

Key Insights and Ideas

  • Surprising Insensitivity to Input Transformations: BERT-based models often fail to detect simple word-level and word-order transformations, maintaining high agreement scores. This suggests they might rely on superficial patterns rather than deep semantic understanding.
  • Chained and Gradient Transformations are More Detectable: When transformations are chained or when gradient-based methods alter critical tokens, models show a greater ability to detect invalid inputs, leading to a dip in performance.
  • Brittle to Adversarial Attacks: The models are highly susceptible to adversarial attacks, with Adv-Rank causing significant shifts in ranking and LSEC leading to substantial performance degradation (e.g., ~30% drop in F1-score).
  • Natural Adversarial Examples are Effective: The LSEC attack, which generates plausible-sounding but incorrect examples, proves to be a potent method for confusing the models.
  • Need for Robustness Evaluation: The study highlights the critical need to evaluate the robustness of biomedical NLP models beyond standard accuracy metrics, especially given their application in sensitive domains.

Key Results

  • Agreement Scores: For individual input transformations, agreement scores often remained high (e.g., >0.85 for many transformations on the NCBI dataset), indicating the model often produced the same (incorrect) prediction.
  • Adversarial Ranking Attack: For a designated embedding shift, MA+ attacks showed considerable shifts in average rank, with performance dips of up to 4-5% for BERT-based ranking models.
  • Least Similar Entity Concatenation (LSEC): This attack significantly degraded performance. For instance, on the NCBI dataset, F1 scores dropped from 0.853 to 0.580 for PubMed BERT and from 0.859 to 0.582 for Bio BERT after the LSEC attack.
  • Varying Transformation Coverage: When 60-80% of test data was transformed, accuracy scores showed noticeable dips, particularly for chained and gradient-based transformations, suggesting the models are better at detecting more complex invalid inputs.

Limitations and Future Work

  • Limitations: The paper acknowledges that while some transformations are detected, the models still struggle with many individual and chained input variations. The adversarial attacks, while effective, are computationally intensive.
  • Future Work: The authors suggest exploring better mitigation strategies to enhance model sensitivity to invalid inputs and developing defense mechanisms to reduce model brittleness against adversarial attacks. They also plan to incorporate more sophisticated adversarial attacks and investigate their impact on different BERT-based normalization models.

Summary updated

BibTeX
@inproceedings{chakraborty2022evaluating,
  author = {Sinchani Chakraborty and Harsh Raj and Srishti Gureja and Tanmay Jain and Atif Hassan and Sayantan Basu},
  title = {{Evaluating the Robustness of Biomedical Concept Normalization}},
  booktitle = {Tl4nlp},
  year = {2022},
}
Figure