TL;DR
No-Harm-Jury is an automated evaluation framework that uses a diverse "jury" of five large language models (LLMs) to audit the safety of medical AI responses. By scoring answers across seven distinct harm dimensions and aggregating results via a robust median, the system identifies "Critical" safety risks that standard accuracy-based benchmarks often overlook.
What Problem Does This Paper Address?
As LLMs achieve human-level performance on medical exams, a dangerous "validity gap" has emerged: high accuracy on benchmarks does not guarantee clinical safety. A model might correctly answer a multiple-choice question but provide a free-text response that contains hallucinations, violates patient autonomy, or ignores privacy.
While human physician review is the gold standard for safety, it is too slow and expensive to scale. Conversely, using a single LLM as a judge introduces idiosyncratic biases. The authors address this by creating a multi-dimensional, multi-model "jury" system that can automatically and scalably evaluate thousands of medical responses for potential real-world harm.
Methods
The framework operates in a rigorous three-stage pipeline:
- Response Generation: A "responder" model (in this study, Ministral-14B) generates answers to 3,000 medical questions.
- Jury Scoring: A panel of five diverse LLM jurors (ranging from 7B to 32B parameters) independently evaluates each response. Each juror provides a score from 0.0 to 1.0 and a text justification for seven harm dimensions: Informational, Social, Psychological, Autonomy, Economic, Privacy, and Epistemic.
- Aggregation and Verdict: The system calculates the median score for each dimension. A response is flagged as Critical if the median score of any single dimension reaches or exceeds a threshold of 0.4.
The use of the median is a key design choice. It provides "Median Robustness," ensuring that the final verdict isn't corrupted by one or two "rogue" jurors who might be overly sensitive or completely blind to a specific type of harm.
Datasets
The authors evaluated 2,849 valid responses across three prominent medical benchmarks:
- PubMedQA: 1,000 expert-annotated biomedical research questions.
- MedQA: 1,000 USMLE-style clinical vignettes (real-world patient scenarios).
- MedMCQA: 1,000 multiple-choice questions from Indian postgraduate entrance exams.
Key Insights and Ideas
- Evaluative Fingerprints: The researchers discovered that LLMs have distinct "personalities" when judging harm. For instance, Gemma3-27B acted as a "universal detector," consistently assigning higher harm scores than its peers. In contrast, Qwen2.5-Coder-7B acted as a "consequences specialist," focusing more on the economic and psychological fallout of bad advice rather than just technical accuracy.
- Complementary Specialization: Diversity in the jury isn't just a safety net; it provides broader coverage. Some models are naturally more sensitive to privacy (the "Privacy Sentinels"), while others are better at catching subtle social biases.
- Accuracy $\neq$ Safety: The results confirm that models can be "smart" but "dangerous." Even on benchmarks where models perform well, the jury found significant rates of potential harm, particularly in clinical vignettes where the stakes of incorrect advice are highest.
- Informational Harm is King: Across all datasets, factual inaccuracies and fabricated evidence (Informational harm) were the primary drivers of "Critical" verdicts.
Key Results
- High Risk in Clinical Scenarios: 13.4% of responses in the MedQA dataset (clinical vignettes) were flagged as Critical. This is significantly higher than the 2.9% flagged in PubMedQA (research abstracts), suggesting that clinical application is a much higher-risk domain for current LLMs.
- Scale and Throughput: The framework successfully executed nearly 100,000 individual scoring operations. Using high-end hardware (NVIDIA H100s), the system averaged about 11.3 seconds to fully evaluate one question across all 35 juror calls (5 jurors × 7 dimensions).
- Bias Suppression: The median aggregation successfully suppressed "global score inflation" from sensitive models like Gemma3 while still preserving the signal needed to catch truly dangerous responses.
Limitations and Future Work
- Single Responder: This study only evaluated responses from one model (Ministral-14B). Future work should compare the "harm profiles" of different responder models.
- The "Gold Standard" Gap: The system currently operates without human-annotated ground truth. While the authors propose a Delphi-style study with medical ethicists to validate the labels, the current "Critical" flags are mechanical and may include false positives.
- Rubric Alignment: The authors noted a slight misalignment between the juror's internal rubric (where 0.5 is "Acceptable") and the system's critical threshold (0.4). Future iterations aim to align these boundaries more tightly.
- Reliability Weighting: The paper introduces a mathematical extension for a "reliability-weighted median," which would give more weight to jurors who historically align better with human experts.
Summary updated