MedRiskEval: Medical Risk Evaluation Benchmark of Language Models, On the Importance of User Perspectives in Healthcare Settings

Jean-Philippe Corbeil, Minseon Kim, Maxime Griot, Sheela Agarwal, Alessandro Sordoni, Francois Beaulieu, Paul Vozila

TL;DR

This paper introduces MedRiskEval, a new benchmark for evaluating the safety of Large Language Models (LLMs) in healthcare, specifically addressing risks from patient, clinician, and general user perspectives. They also release PatientSafetyBench, a novel dataset focused on patient-centric risks, finding that even advanced LLMs exhibit significant vulnerabilities in medical contexts.

What Problem Does This Paper Address?

The increasing adoption of LLMs in healthcare presents a critical challenge: while general safety benchmarks exist, they often fall short when applied to specialized domains like medicine. In healthcare, LLMs can be used by a diverse range of users, from patients with no medical expertise to seasoned clinicians. The outputs of these models can have direct and serious consequences for human health. Existing medical risk evaluations have largely focused on clinician perspectives, neglecting the unique risks and expectations of patients. This gap leaves a significant vulnerability in deploying LLMs safely in real-world medical scenarios, particularly for non-expert users.

Methods

The core of this work is the development of MedRiskEval, a comprehensive medical risk evaluation benchmark. MedRiskEval is structured around three user perspectives:

  • Patient: This perspective is addressed by the newly introduced PatientSafetyBench (PSB) dataset.
  • Clinician: This perspective is covered by leveraging the existing MedSafetyBench dataset.
  • General User: This perspective is assessed using general safety benchmarks like XSTest, JBB, and FACTS.

The authors also introduce PatientSafetyBench (PSB), a novel dataset designed to fill the gap in patient-centric risk evaluation. PSB contains 466 synthetic samples, validated by medical doctors, across five critical risk categories: Harmful or Dangerous Medical Advice, Misdiagnosis and Overconfidence, Unlicensed Practice of Medicine, Health Misinformation, and Discrimination in Medical Context. The generation process involved prompting GPT-4 with risk definitions and then using OpenAI's 03-mini as an LLM judge to score the quality of generated queries. A subset was further reviewed by medical doctors to ensure relevance, severity, and harm likelihood.

Datasets

  • PatientSafetyBench (PSB) (New): A dataset of 466 patient-focused risk scenarios, covering 5 critical risk categories. Validated by medical doctors.
  • MedSafetyBench (Han et al., 2024): Used to evaluate clinician-centric risks.
  • XSTest (Röttger et al., 2024): A general safety benchmark for harmfulness.
  • JBB (Chao et al., 2024): Used to evaluate jailbreaking.
  • FACTS (Jacovi et al., 2025): Used to evaluate groundedness in a medical context.

Key Insights and Ideas

  • Patient perspective is crucial and underserved: The creation of PatientSafetyBench highlights that existing evaluations are heavily skewed towards clinician views, leaving a critical gap in understanding patient-specific risks.
  • Medical fine-tuning doesn't guarantee safety: Surprisingly, even models specifically fine-tuned for medical applications (like the MediPhi collection) and top-tier general models (like GPT-4.1) still exhibit significant safety vulnerabilities, particularly in patient-facing scenarios.
  • Role-dependent vulnerabilities: LLMs can behave differently depending on the user's perspective (patient vs. clinician), and these role-specific vulnerabilities are not well-captured by general benchmarks.
  • Misdiagnosis and overconfidence are major concerns: Patients, lacking medical expertise, are particularly vulnerable to LLMs that exhibit overconfidence or provide speculative diagnostic information.
  • Harmful advice and unlicensed practice are high-risk categories: These areas, where LLMs might directly suggest dangerous actions or impersonate medical professionals, pose significant threats.

Key Results

  • Widespread vulnerabilities: Nearly a third of all answers from evaluated models were categorized as "unsafeguarded engagement" or worse on the PatientSafetyBench, with the exception of the GPT-4.1 family.
  • GPT-4.1 shows better, but not perfect, performance: While the GPT-4.1 family generally performed better, even they achieved at most a refusal rate of 58.2% on patient-oriented risks.
  • Medical variants show mixed results: While some medical fine-tuning improved refusal rates, it didn't always translate to better overall safety, and in some cases, like MP-Instruct, it led to an increase in harmful responses.
  • Clinician risks (MedSafetyBench) show higher refusal rates: Models tended to refuse more often on clinician-focused prompts, suggesting a different safety mechanism activation compared to patient-focused queries.
  • General safety benchmarks show similar trends: For harmfulness and jailbreaking, models generally performed similarly, with some minor degradations observed for medical variants.

Limitations and Future Work

  • Dataset size: PatientSafetyBench, while validated, is relatively small (466 samples).
  • Single-turn queries: The current benchmark focuses on short, single-turn interactions, not multi-turn conversations or long-context clinical scenarios.
  • Limited scope of risks: The taxonomy doesn't cover all potential safety concerns, such as privacy, legal, and regulatory compliance.
  • English-centric: The evaluation is limited to English, restricting generalization to other languages.
  • Operationalizing risk: The current analysis relies on strict refusal behavior. Future work should explore risk-benefit evaluations that balance utility with safety.
  • Reproducibility: For proprietary models, exact replication of results can be challenging due to rapid evolution and version deprecation.

The authors suggest future work should extend to multi-turn conversations, explore a broader range of safety concerns, and investigate risk-benefit trade-offs for safer LLM deployment in healthcare.

Summary updated

Abstract

As the performance of large language models (LLMs) continues to advance, their adoption in the medical domain is increasing. However, most existing risk evaluations largely focused on general safety benchmarks. In the medical applications, LLMs may be used by a wide range of users, ranging from general users and patients to clinicians, with diverse levels of expertise and the model’s outputs can have a direct impact on human health which raises serious safety concerns. In this paper, we introduce MedRiskEval, a medical risk evaluation benchmark tailored to the medical domain. To fill the gap in previous benchmarks that only focused on the clinician perspective, we introduce a new patient-oriented dataset called PatientSafetyBench containing 466 samples across 5 critical risk categories. Leveraging our new benchmark alongside existing datasets, we evaluate a variety of open-and closed-source LLMs. To the best of our knowledge, this work establishes an initial foundation for safer deployment of LLMs in healthcare.
BibTeX
@inproceedings{corbeil2026medriskeval,
  author = {Jean-Philippe Corbeil and Minseon Kim and Maxime Griot and Sheela Agarwal and Alessandro Sordoni and Francois Beaulieu and Paul Vozila},
  title = {{MedRiskEval: Medical Risk Evaluation Benchmark of Language Models, On the Importance of User Perspectives in Healthcare Settings}},
  booktitle = {EACL (Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.39},
}
Figure