Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks

Jack Gallifant, Shan Chen, Pedro Moreira, Nikolaj Munch, Mingye Gao, Jackson Pond, Leo Anthony Celi, Hugo J. W. L. Aerts, Thomas Hartvigsen, Danielle S. Bitterman

Recent advancements in Large Language Models (LLMs) have led to impressive scores on medical licensing exams, suggesting a potential for clinical utility. However, new research titled "Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks" uncovers a significant vulnerability: these models often rely on memorization rather than true clinical reasoning. By introducing the RABBITS benchmark, researchers demonstrate that LLMs are remarkably sensitive to simple nomenclature changes, such as swapping a generic drug name for its brand-name equivalent.

The Problem: Reasoning vs. Pattern Matching

In clinical practice, synonymy is pervasive. A physician might prescribe acetaminophen, while a patient reports taking Tylenol. For an LLM to be safe and effective, it must recognize these terms as interchangeable. Current benchmarks like MedQA are often used to validate LLM performance, but they may suffer from a "Clever Hans" effect—where models appear intelligent because they have memorized specific test questions during training rather than understanding the underlying medical concepts.

Methodology: The RABBITS Pipeline

To test this hypothesis, the researchers developed a systematic pipeline to evaluate the robustness of LLMs to drug name substitutions. This process, which resulted in the RABBITS (Robust Assessment of Biomedical Benchmarks Involving drug Term Substitutions) benchmark, involves four key stages:

The RABBITS pipeline: Extracting drug pairs from RxNorm, inspecting for ambiguity, swapping names in benchmarks, and expert physician verification.
  1. Extract: Identifying brand-generic pairs using the RxNorm ontology.
  2. Inspect: Removing ambiguous drugs or those with multiple ingredients to ensure clean substitutions.
  3. Swap: Using regular expressions to replace drug names within existing benchmarks (MedQA and MedMCQA).
  4. Verify: Physician experts manually review the swapped data points to ensure medical accuracy and context preservation.

Key Results: The Fragility of LLMs

The evaluation of a wide array of models—ranging from open-source (Llama-3, Qwen) to proprietary (GPT-4, Claude 3, Gemini)—revealed a consistent performance gap. Swapping generic names for brand names caused an average 4% drop in accuracy across all models, with some dips reaching as high as 10%.

Accuracy comparison showing a consistent performance drop when drug names are swapped from generic to brand names across various LLMs.

Interestingly, while models are excellent at identifying drug pairs in isolation (e.g., knowing that Advil is ibuprofen), they struggle to apply this knowledge within complex clinical vignettes. This suggests a "Knowledge vs. Application" gap.

The Role of Data Contamination

A critical finding of the study is the high level of data contamination in existing benchmarks. Using Infini-gram to search pre-training datasets like Dolma and C4, the researchers found that up to 99% of MedQA test questions were present in common training data.

Models showed a larger performance drop on the less-contaminated MedMCQA dataset when names were swapped. This indicates that high scores on benchmarks like MedQA are often driven by the model "recalling" a specific string of text it has seen before. When that string is altered by a drug name swap, the memorized pattern is broken, and the model's performance collapses.

Conclusion and Future Directions

The study concludes that scaling models alone does not solve this fragility. Even massive models like Llama-3-70B experienced significant accuracy drops. Future medical AI evaluation must move toward dynamic benchmarks that test a model's ability to handle the linguistic variability inherent in real-world healthcare, rather than relying on static, potentially leaked question sets.

The RABBITS leaderboard is now available on HuggingFace, offering a more rigorous standard for evaluating the true clinical reasoning capabilities of Large Language Models.

Summary updated

Abstract

Medical knowledge is context-dependent and requires consistent reasoning across various natural language expressions of semantically equivalent phrases. This is particularly crucial for drug names, where patients often use brand names like Advil or Tylenol instead of their generic equivalents. To study this, we create a new robustness dataset, RABBITS, to evaluate performance differences on medical benchmarks after swapping brand and generic drug names using physician expert annotations. We assess both open-source and API-based LLMs on MedQA and MedMCQA, revealing a consistent performance drop ranging from 1-10\%. Furthermore, we identify a potential source of this fragility as the contamination of test data in widely used pre-training datasets. All code is accessible at https://github.com/BittermanLab/RABBITS, and a HuggingFace leaderboard is available at https://huggingface.co/spaces/AIM-Harvard/rabbits-leaderboard.
BibTeX
@inproceedings{gallifant2024language,
  author = {Jack Gallifant and Shan Chen and Pedro Moreira and Nikolaj Munch and Mingye Gao and Jackson Pond and Leo Anthony Celi and Hugo J. W. L. Aerts and Thomas Hartvigsen and Danielle S. Bitterman},
  title = {{Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.726},
}
Figure