MedGuardBench: A Patient-Oriented Benchmark for Personalized Guardrail Development

Sara Khosravi, Xiaofei Wen, Majid Sarrafzadeh, Muhao ChenarXiv 2026 · arXiv

TL;DR

MedGuardBench introduces a new frontier for LLM safety: personalized guardrails. By shifting from "one-size-fits-all" content filtering to patient-conditioned medication safety, the authors demonstrate that guardrails can learn to judge a prescription's safety based on a specific individual's medical profile, significantly reducing both dangerous misses and unnecessary "over-blocking."

What Problem Does This Paper Address?

Traditional safety guardrails are "content-only." They look for keywords or patterns associated with harm and block them. However, in high-stakes domains like medicine, safety is personal. A 400mg dose of ibuprofen is perfectly safe for a healthy adult but potentially lethal for a patient with renal impairment or someone taking warfarin.

Current medical LLMs might "know" drug facts, but they struggle to perform a rigorous check against a patient's specific details. Content-only guardrails either miss real hazards (false negatives) or block safe, necessary actions (false positives/over-blocking). This paper tackles the challenge of building guardrails that condition their safety verdict on the individual being affected.

Methods

The authors propose a three-pronged approach to training "personalized reasoning guardrails":

  1. Personalized Reasoning Distillation (SFT): Instead of just training on "Safe/Unsafe" labels, the models are trained on detailed reasoning traces from a "teacher" model (DeepSeek-V4-Pro). These traces explicitly link specific patient attributes (e.g., "Age: 72") to clinical risks.
  2. Context Unrolling (CU): This is a novel multi-stage generation technique. Before the model gives a final verdict, it is forced to write out a field-by-field safety check across 17 risk categories. By making the model "show its work" for every patient attribute, the authors found they could significantly reduce "shotgun" flagging—where a model calls something unsafe just because it sees any abnormality in a profile.
  3. Guided GRPO: The authors adapt Group Relative Policy Optimization (a form of reinforcement learning) using a rule-based reward. This reward specifically prioritizes catching rare, complex risks (like multi-step drug-allergy links) that standard fine-tuning might miss.
Figure 1: Overview of MedGuardBench. 5,001 scenarios over 87 common medications are generated from Drugs.com and DrugBank monographs, audited by a teacher LLM, and human-checked. Guardrails are then trained with Personalized Reasoning Distillation (SFT), then two follow-up steps applied on their own from the SFT checkpoint—Context Unrolling and Guided GRPO—and tested with and without the patient profile.

Datasets

The paper introduces MedGuardBench, a benchmark consisting of 5,001 clinical scenarios covering 87 common medications.

  • Grounded in Evidence: Scenarios are generated based on trusted sources like Drugs.com and DrugBank.
  • Drug-Disjoint Splits: To test true generalization, the medications in the test set are entirely unseen during training.
  • Real-World Validation: Beyond the synthetic benchmark, the models were evaluated on four real-patient cohorts (FAERS, NHANES, MIMIC-IV, and DDInter) to ensure the learned safety rules transfer to actual clinical data.

Key Insights and Ideas

  • Prompting isn't enough: Base LLMs and even medically pre-trained models (like MedGemma) barely use patient profiles when prompted zero-shot. They tend to either ignore the profile or over-block everything once a profile is present. Effective personalization must be learned.
  • Evidence must precede the verdict: The "Context Unrolling" method proved that forcing a model to justify a risk flag before raising it is the most effective way to cut down on false alarms.
  • Sensitivity to content, not just presence: Through "profile swapping" experiments, the authors proved their models aren't just reacting to the existence of a medical profile; they are actually reading the specific values. When a profile is shuffled or removed, the model's performance collapses, whereas zero-shot models remain largely unchanged.
Figure 3: Radar view of test-time profile pertur- bations. Rows are Qwen models, columns are train- ing arms; each panel overlays the four test conditions (True Profile, No Profile, Shuffled Profile, and Non- Personalized) across all six metrics: verdict accuracy (Acc), unsafe recall (Rec), false-positive rate on safe prescriptions (FPR), 17-category micro-F1 and macro- F1 (Mic/Mac), and 17-category exact match (EM). Axes are oriented so that farther from the center is better (FPR shown as 100 −FPR). The personalized arms (SFT, CU, and GRPO) contract sharply when the profile is re- moved or shuffled, while the zero-shot arm stays stable.

Key Results

  • Accuracy Boost: Personalized guardrails outperformed matched "no-profile" controls by 12.0 to 20.0 accuracy points.
  • High Recall: The Guided GRPO models achieved 96%–99% unsafe recall on most backbones, successfully catching nearly all dangerous prescriptions.
  • Reduced Over-blocking: Context Unrolling (CU) was the most reliable method for reducing false positives, achieving the highest overall verdict accuracy and exact match for risk categories.
  • Real-World Transfer: On real-patient data, the trained models identified correct hazards at a rate of 76%–93%, compared to near-zero attribution for general-purpose zero-shot baselines.
Figure 2: Safety operating points on the drug-disjoint test set. Colors indicate experimental arms and mark- ers denote backbones. The target region requires recall ≥95% and FPR < 20%. Key takeaway: Personalized models achieve high unsafe recall while limiting false alarms; removing patient context generally shifts mod- els toward over-blocking or reduced recall.

Limitations and Future Work

The authors are careful to note that these models are research prototypes and not certified medical devices.

  • Structured Scope: The benchmark uses clean, 20-field tables. Real-world Electronic Health Records (EHRs) are noisy and often unstructured.
  • Rare Risks: Some risk categories (like Caffeine Intake) have very few samples, leading to high variance in performance for those specific risks.
  • The "Last Mile": Even the best models still miss ~2-4% of unsafe prescriptions and falsely flag ~7-12% of safe ones. Future work is needed to calibrate these models further and integrate them into human-in-the-loop clinical workflows.

Summary updated

Abstract

Safety is personal: the same action can be safe for one person but unsafe for another. Yet current guardrails judge content without considering who is affected. Using medication safety, we introduce \textit{patient-conditioned medication guarding}, which predicts a safety verdict and 17 risk categories from a patient profile and prescription. We release \textbf{MedGuardBench}, a drug-disjoint benchmark of 5{,}001 scenarios across 87 medications. We find that standard guardrails rely on surface cues, such as wording and profile presence, rather than patient facts. To address this, we develop \textit{personalized reasoning guardrails} with three components. \textbf{Personalized Reasoning Distillation} links risks to relevant patient fields. \textbf{Context Unrolling (CU)} makes models check each field themselves, reducing false alarms on most backbones; controls show that providing the check does not yield the same benefit. A \textbf{rule-based reward (guided GRPO)} improves unsafe recall and complex-risk detection, including allergies and adverse reactions, on some backbones, but increases false alarms on others. Our models outperform base models and matched no-profile controls in accuracy, while reducing false alarms relative to no-profile controls. Profile removal and swapping confirm that gains come from patient information, not fine-tuning alone.
BibTeX
@misc{khosravi2026medguardbench,
  author = {Sara Khosravi and Xiaofei Wen and Majid Sarrafzadeh and Muhao Chen},
  title = {{MedGuardBench: A Patient-Oriented Benchmark for Personalized Guardrail Development}},
  year = {2026},
  eprint = {2611.22069},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22069},
}
Figure