What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

Betty Van Aken, Sebastian Herrmann, Alexander LöserNAACL 2022

TL;DR

This paper introduces a novel framework for "behavioral testing" of clinical NLP models, designed to reveal how subtle changes in patient descriptions (like gender or ethnicity) impact model predictions. They demonstrate that even models with similar performance metrics can exhibit drastically different, and sometimes harmful, biases, highlighting the need for more transparent model evaluation in healthcare.

What Problem Does This Paper Address?

Clinical NLP models hold immense promise for improving patient care by analyzing vast amounts of medical text to identify risks and aid decision-making. However, these models, often built on complex deep neural networks, can be "black boxes." This opacity makes it difficult to understand why they make certain predictions. Previous research has shown that these models can inherit and even amplify systemic biases present in the data, leading to potentially unfair or inaccurate outcomes, especially for minority groups. Furthermore, their behavior with out-of-distribution data can be unpredictable. It's crucial to understand these learned patterns and their influence on model decisions before deploying them in clinical practice, where errors can have serious consequences.

Methods

The authors propose a "behavioral testing framework" specifically for clinical NLP models. This framework works by taking an existing dataset of clinical notes and systematically altering specific tokens related to patient characteristics. For example, they might change "man" to "woman" or alter pronouns. They then analyze how the model's predictions change for these altered samples compared to the original.

Key aspects of their method include:

  • Sample Alterations: Creating distinct "test groups" by modifying only one patient characteristic at a time (e.g., age, gender, ethnicity) while keeping the rest of the clinical note identical. This allows them to isolate the influence of that specific characteristic.
  • Prediction Analysis: Instead of comparing predictions to ground truth (which can be problematic due to inherent biases in clinical data), they focus on detecting how the model's outputs change. This approach is more robust to noisy or biased labels and allows for the evaluation of rare mentions.
  • Model Agnostic: The framework is designed to work with any text-based outcome prediction model, not just specific architectures.

Datasets

The primary dataset used for this study is MIMIC-III (Medical Information Mart for Intensive Care III). They utilize the outcome prediction task setup previously defined by van Aken et al. (2021), which includes 48,745 English admission notes annotated with patient outcomes at discharge. Specifically, they focus on the outcomes of "diagnoses at discharge" and "in-hospital mortality." The analysis is conducted on a test set of 9,829 samples from this setup. No new benchmarks were introduced; they leveraged existing data and task definitions.

Key Insights and Ideas

  • Performance Metrics Can Be Deceiving: Models with similar AUROC scores can exhibit vastly different internal behaviors and biases. This highlights the inadequacy of relying solely on aggregate performance metrics for evaluating clinical AI.
  • Subtle Textual Changes Have Big Impacts: Even minor alterations to patient descriptions (e.g., changing gender pronouns) can significantly shift model predictions, sometimes in medically implausible ways.
  • "Black Box" Models Inherit and Amplify Societal Biases: The models demonstrate a tendency to reflect and sometimes exaggerate existing biases related to gender and ethnicity found in the training data.
  • Transgender Patients Face Unique Risks: Models showed a consistent pattern of assigning lower mortality risk to transgender patients, which could be due to their relative youth in the dataset but may lead to underestimation of risks in real-world scenarios.
  • Ethnicity Mentions Can Lead to Harmful Underestimation: For African American patients, some models showed a pattern of assigning lower diagnostic probabilities, potentially leading to fewer tests and disadvantages in treatment.

Key Results

The paper analyzes three state-of-the-art clinical NLP models (PubMedBERT, CORe, and BioBERT) fine-tuned on the MIMIC-III dataset.

  • Gender: All models predicted lower mortality risk for transgender patients compared to male or female patients. PubMedBERT showed particularly high sensitivity to gender mentions, often predicting lower probabilities for diagnoses when "transgender" was mentioned.
  • Age: Models generally followed patterns observed in the training data regarding age and diagnosis risk, with some diagnoses increasing and others decreasing with age. However, they sometimes showed "peaks" indicating a lack of nuanced understanding of numerical progression.
  • Ethnicity: Mentioning any ethnicity generally decreased predicted mortality risk across all models. PubMedBERT was highly sensitive to ethnicity mentions, predicting certain diagnoses more or less likely based on the mentioned ethnicity. CORe and BioBERT showed a potentially harmful pattern of assigning lower diagnostic risk to African American patients.

Limitations and Future Work

Limitations:

  • Interpreting "Correct" vs. "Incorrect" Behavior: The framework doesn't definitively label behavior as correct or false. Increased risk allocation could be beneficial (more tests) or harmful (mistreatment). The impact of rare mentions (like transgender) is also debated – should they have less influence due to low frequency, or are they important signals?
  • Model Agnosticism vs. Specificity: While the framework is model-agnostic, understanding specific architectural or pre-training influences requires careful comparison.

Future Work:

  • Intersectionality: The framework can be extended to test model behavior regarding combinations of patient characteristics (e.g., older, Black, female patients).
  • Iterative Model Fine-tuning with Medical Professionals: Involving clinicians in the loop during model development and fine-tuning is proposed as a promising direction to guide models towards medically meaningful patterns and away from harmful ones.
  • Model Cards for Clinical NLP: The authors suggest using behavioral analysis results as part of "model cards" to communicate model behavior transparently to medical professionals, similar to proposals for general AI models.

Summary updated

BibTeX
@inproceedings{aken2022what,
  author = {Betty Van Aken and Sebastian Herrmann and Alexander Löser},
  title = {{What Do You See in this Patient? Behavioral Testing of Clinical NLP Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure