Recent advancements in Large Language Models (LLMs) have led to impressive scores on medical licensing exams, suggesting a potential for clinical utility. However, new research titled "Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks" uncovers a significant vulnerability: these models often rely on memorization rather than true clinical reasoning. By introducing the RABBITS benchmark, researchers demonstrate that LLMs are remarkably sensitive to simple nomenclature changes, such as swapping a generic drug name for its brand-name equivalent.
The Problem: Reasoning vs. Pattern Matching
In clinical practice, synonymy is pervasive. A physician might prescribe acetaminophen, while a patient reports taking Tylenol. For an LLM to be safe and effective, it must recognize these terms as interchangeable. Current benchmarks like MedQA are often used to validate LLM performance, but they may suffer from a "Clever Hans" effect—where models appear intelligent because they have memorized specific test questions during training rather than understanding the underlying medical concepts.
Methodology: The RABBITS Pipeline
To test this hypothesis, the researchers developed a systematic pipeline to evaluate the robustness of LLMs to drug name substitutions. This process, which resulted in the RABBITS (Robust Assessment of Biomedical Benchmarks Involving drug Term Substitutions) benchmark, involves four key stages:
- Extract: Identifying brand-generic pairs using the RxNorm ontology.
- Inspect: Removing ambiguous drugs or those with multiple ingredients to ensure clean substitutions.
- Swap: Using regular expressions to replace drug names within existing benchmarks (MedQA and MedMCQA).
- Verify: Physician experts manually review the swapped data points to ensure medical accuracy and context preservation.
Key Results: The Fragility of LLMs
The evaluation of a wide array of models—ranging from open-source (Llama-3, Qwen) to proprietary (GPT-4, Claude 3, Gemini)—revealed a consistent performance gap. Swapping generic names for brand names caused an average 4% drop in accuracy across all models, with some dips reaching as high as 10%.
Interestingly, while models are excellent at identifying drug pairs in isolation (e.g., knowing that Advil is ibuprofen), they struggle to apply this knowledge within complex clinical vignettes. This suggests a "Knowledge vs. Application" gap.
The Role of Data Contamination
A critical finding of the study is the high level of data contamination in existing benchmarks. Using Infini-gram to search pre-training datasets like Dolma and C4, the researchers found that up to 99% of MedQA test questions were present in common training data.
Models showed a larger performance drop on the less-contaminated MedMCQA dataset when names were swapped. This indicates that high scores on benchmarks like MedQA are often driven by the model "recalling" a specific string of text it has seen before. When that string is altered by a drug name swap, the memorized pattern is broken, and the model's performance collapses.
Conclusion and Future Directions
The study concludes that scaling models alone does not solve this fragility. Even massive models like Llama-3-70B experienced significant accuracy drops. Future medical AI evaluation must move toward dynamic benchmarks that test a model's ability to handle the linguistic variability inherent in real-world healthcare, rather than relying on static, potentially leaked question sets.
The RABBITS leaderboard is now available on HuggingFace, offering a more rigorous standard for evaluating the true clinical reasoning capabilities of Large Language Models.
Summary updated