Evaluating the Robustness of Adverse Drug Event Classification Models using Templates

Dorothea MacPhail, David Harbecke, Lisa Raithel, Sebastian MöllerACL 2024

TL;DR

This paper introduces a novel template-based evaluation method to assess the robustness of Adverse Drug Event (ADE) classification models beyond standard accuracy metrics. By creating handcrafted test cases for specific linguistic capabilities like temporal order and negation, they reveal that models with similar overall performance can exhibit significant differences in their understanding of these crucial nuances, highlighting potential blind spots in current evaluation practices.

What Problem Does This Paper Address?

Detecting Adverse Drug Events (ADEs) from user-generated text, like social media posts, is critical for public health. While models have shown impressive performance on standard benchmarks, simply achieving high accuracy on held-out test sets doesn't guarantee a deep understanding of the underlying linguistic phenomena. Models might rely on superficial heuristics or "shortcuts" rather than truly grasping concepts like temporal relationships (e.g., did the ADE occur before or after taking the drug?), negation, or sentiment. This is particularly concerning in high-stakes domains like medicine, where misinterpretations can have serious consequences. The paper aims to address this gap by proposing a more thorough evaluation method that probes specific linguistic capabilities.

Methods

The core contribution is a template-based evaluation framework inspired by the CheckList methodology. The authors manually crafted 99 base templates with 1505 variations, designed to test four specific linguistic capabilities:

  • Temporal Order: Assessing if models can correctly interpret the sequence of events (drug intake vs. ADE occurrence), including variations with single and double time entities.
  • Positive Sentiment: Evaluating if models can identify ADEs even when reported with positive sentiment (e.g., relief from original symptoms alongside a mild ADE).
  • Beneficial Effect: Testing the model's ability to distinguish between a true ADE and a secondary effect that is beneficial to the patient.
  • Negation: Probing the model's understanding of negated statements related to ADEs.

These templates contain placeholders for drugs and ADEs, which are then filled with entities extracted from various datasets. The filled templates form test cases that are fed to fine-tuned ADE classification models. The performance on these template-based tests is then compared to the models' performance on a standard held-out test set.

Datasets

The paper uses a custom dataset for fine-tuning the models, which is a merged corpus comprising:

  • SMM4H-2021 Shared Task 1a training data: Social media posts (Twitter, DailyStrength) annotated for ADEs.
  • SMM4H-2017 Shared Task dataset: Twitter posts annotated for ADEs.
  • NADE dataset: Artificially negated tweets derived from SMM4H-2019 data.

For filling the template placeholders, the authors utilized:

  • PsyTAR dataset: Patient reports on psychiatric medications, used to extract drug names and ADE entities.
  • Ask-a-Patients: A patient forum, also used for extracting drug names and ADE entities.

The template-based test cases themselves constitute a newly curated benchmark for evaluating specific linguistic capabilities in ADE detection.

Key Insights and Ideas

  • Standard metrics can be misleading: Models that perform similarly on held-out test sets can exhibit vastly different robustness when tested on specific linguistic phenomena. This highlights the limitations of relying solely on traditional evaluation metrics.
  • Models struggle with nuance: Even advanced transformer models show significant weaknesses in understanding temporal order and distinguishing between ADEs and beneficial effects. This suggests they might be learning superficial patterns rather than true causal or semantic relationships.
  • Negation is a strong suit (relatively): Models performed best on negation tests, indicating that this is a more well-understood linguistic challenge for them compared to temporal order or beneficial effects.
  • Beneficial effects are a major blind spot: Models consistently misclassified beneficial effects as ADEs, suggesting a failure to grasp the context that differentiates a negative outcome from a positive secondary effect.
  • Template design matters: The performance variations across different template variations for the same capability indicate that the specific wording and structure of test cases can significantly influence model predictions.

Key Results

The paper evaluated two transformer-based models: BioRedditBERT (a medically-focused model) and XLM-ROBERTa (a general multilingual model).

  • Baseline Performance: Both models achieved similar F1-scores (around 0.700) on the positive class (ADE) in the held-out test set, indicating comparable overall performance.
  • Template-Based Performance:
    • Negation: Both models performed well, with BioRedditBERT achieving 92% and XLM-ROBERTa 94% recall.
  • Beneficial Effect: Performance was strikingly poor. Models correctly classified only 7.5% (BioRedditBERT) and 5.8% (XLM-ROBERTa) of the negative class (beneficial effect) test cases, often misclassifying them as ADEs.
  • Temporal Order: Performance varied significantly. For the negative class, recall ranged from 54%-78%. For the positive class, models performed poorly on standard and double time entity tests (25%-48%) but exceeded baseline on single time entity tests (80%-90%).
  • Positive Sentiment: Models performed above baseline on positive sentiment test cases (72% for BioRedditBERT, 68% for XLM-ROBERTa).

Limitations and Future Work

  • Unnatural Phrases: The automatic filling of templates can sometimes create unnatural or biased language, which might affect model performance.
  • Limited Linguistic Variety: The templates, while diverse, might not capture the full spectrum of language used in social media, potentially introducing bias.
  • Feature Omission: The templates did not incorporate all aspects of social media text, such as usernames, hashtags, or non-standard punctuation, which could lead to a slight mismatch with real-world data.

Future work could involve expanding the set of capabilities tested, evaluating more models, and using large language models to generate more diverse templates. The authors also suggest incorporating insights from template-based evaluation to improve model training, potentially by including challenging test cases as new training data.

Summary updated

BibTeX
@inproceedings{macphail2024evaluating,
  author = {Dorothea MacPhail and David Harbecke and Lisa Raithel and Sebastian Möller},
  title = {{Evaluating the Robustness of Adverse Drug Event Classification Models using Templates}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure