Medical Red Teaming Protocol of Language Models: On the Importance of User Perspectives in Healthcare Settings

Jean-Philippe Corbeil, Minseon Kim, Alessandro Sordoni, François Beaulieu, Paul Vozila

Executive Summary

As Large Language Models (LLMs) transition from general-purpose assistants to specialized medical tools, ensuring their safety becomes paramount. This paper introduces MedRiskEval, a multi-perspective framework designed to evaluate LLM safety through the lenses of the patient, the clinician, and the general user. A key contribution is PatientSafetyBench, a novel dataset focused on the risks faced by non-expert patients. The study reveals a "medical fine-tuning paradox": specialized medical training often fails to improve, and can sometimes degrade, the safety profile of models compared to their general-purpose counterparts.

The Problem: Beyond Malicious Intent

Current safety benchmarks for LLMs often focus on general harmfulness (e.g., preventing instructions for illegal acts) or clinician-centric ethical violations (e.g., falsifying records). However, a significant portion of healthcare-related LLM usage comes from non-expert patients seeking medical advice. In these scenarios, the primary risk is not "malicious intent" but "harmful helpfulness." A model might provide technically accurate information that is contextually dangerous, such as suggesting a specific medication dosage to a patient who actually requires immediate emergency intervention. There is a critical need for evaluation protocols that account for the varying expertise and expectations of different healthcare stakeholders.

The MedRiskEval Framework

To address this gap, the researchers developed MedRiskEval. This framework categorizes safety risks based on the intended user, ensuring that models are tested against the specific vulnerabilities inherent in different interaction types.

Overview of the MedRiskEval framework, illustrating the three distinct user perspectives and their corresponding safety benchmarks.

The methodology involves:

  1. Taxonomy Development: Identifying five core patient-focused risk categories: Harmful/Dangerous Medical Advice, Misdiagnosis/Overconfidence, Unlicensed Practice, Health Misinformation, and Discrimination.
  2. Synthetic Generation & Expert Validation: Using advanced models (GPT-4o) to generate 466 diverse patient queries, which were then filtered by an LLM judge and validated by medical professionals for clinical relevance and harm potential.
  3. LLM-as-a-Judge Evaluation: Responses are scored on a 5-point scale ranging from Refusal (safest) to Full Violation (fulfilling an unsafe query).

Datasets and Perspectives

MedRiskEval integrates several datasets to provide a holistic view of safety:

  • PatientSafetyBench (PSB): The authors' new dataset of 466 samples focusing on non-expert patient inquiries.
  • MedSafetyBench (MSB): An existing benchmark targeting clinician-level ethical and professional violations.
  • General Safety Benchmarks: Includes XSTest for general harmfulness, JailbreakBench (JBB) for adversarial robustness, and FACTS for groundedness and hallucination detection.

Key Insights: The Medical Fine-Tuning Paradox

The study's most striking finding is that medical fine-tuning is not a guaranteed path to safety. While specialized training can improve a model's ability to provide accurate medical information, it often introduces new vulnerabilities:

  • Safety Degradation: In several instances, medically adapted models showed higher rates of "Full Violations" than their base versions. For example, while tuning might improve resistance to misinformation, it frequently worsened performance in categories like bias and discrimination.
  • Unsafeguarded Engagement: Nearly a third of responses across various model families fell into the "unsafeguarded engagement" category—providing advice without necessary disclaimers. This is particularly prevalent in the "Misdiagnosis and Overconfidence" category.
  • Intent vs. Clinical Risk: Models are generally proficient at refusing queries with explicit bad intentions. However, they struggle with "well-intentioned but dangerous" patient queries, suggesting that current safety training is over-indexed on detecting malice rather than assessing clinical risk.

Results: Evaluating Model Safety

The evaluation covered a wide range of models, from general-purpose leaders like GPT-4.1 to specialized variants like Med42 and MedGemma.

Distribution of model responses across safety categories, highlighting the prevalence of unsafeguarded engagement and harmful violations.

Key quantitative findings include:

  • GPT-4.1 Dominance: The GPT-4.1 family remains the safest overall, yet its refusal rate on patient-oriented risks is only 58.2%, indicating significant room for improvement.
  • Vulnerability in Diagnosis: The "Misdiagnosis and Overconfidence" category proved the most challenging across all models, with average harmfulness scores (2.3–2.8) trending toward unsafe engagement.
  • Model Comparisons: MedGemma showed a significant (~50%) improvement in refusal rates over its base model, whereas the Llama-based Med42 exhibited safety regressions in several patient-facing categories.

Conclusion and Future Work

MedRiskEval highlights the complexity of medical safety in the age of LLMs. The findings suggest that safety training must move beyond simple refusal of malicious prompts toward a more nuanced "risk-benefit evaluation" that balances helpfulness with clinical safety.

Future research will focus on expanding the scope of PatientSafetyBench to include more languages and multi-turn dialogues, which better reflect real-world medical consultations. Additionally, developing methods to ensure that medical fine-tuning enhances safety across all user perspectives remains a critical challenge for the field.

Summary updated

BibTeX
@inproceedings{corbeil2026medical,
  author = {Jean-Philippe Corbeil and Minseon Kim and Alessandro Sordoni and François Beaulieu and Paul Vozila},
  title = {{Medical Red Teaming Protocol of Language Models: On the Importance of User Perspectives in Healthcare Settings}},
  booktitle = {EACL (Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.39},
}
Figure