Risk-graded Safety for Handling Medical Queries in Conversational AI

Gavin Abercrombie, Verena Rieser

Abstract

As users increasingly turn to Conversational AI (ConvAI) for medical advice, the need for robust safety mechanisms grows. Traditional safety research focuses on toxic content, but medical misinformation presents a unique "silent" danger. This paper introduces a risk-graded framework for evaluating medical queries and responses, moving beyond binary safety labels. Our findings indicate that while automated systems and crowdworkers can identify medical topics, they struggle to assess the severity of life-threatening situations, often underestimating risks that require expert clinical judgment.

Introduction: The Imposter Effect

Large Language Models (LLMs) and chatbots are becoming ubiquitous, yet their application in healthcare is fraught with risk. A primary concern is the Imposter Effect, where a system provides factually structured but clinically disastrous advice—such as suggesting it is safe to mix contraindicated medications.

Existing safety research often treats safety as a binary (safe vs. unsafe). However, medical safety is inherently nuanced; a query about a minor injury requires a different response strategy than a query regarding a heart attack. This study explores whether we can reliably automate the detection and grading of medical risk and evaluates the reliability of non-expert crowdworkers compared to medical professionals.

Methodology: A Risk-Graded Framework

The researchers developed a multi-stage approach to evaluate risk using a framework adapted from the World Economic Forum (WEF). This scheme grades risk on a scale from I (Low) to IV (Very High), where Level I involves general information and Level IV involves specific treatment plans or diagnoses.

The risk-graded framework mapping query seriousness to response risk levels (I-IV).

To test this framework, the authors created a corpus of 5,125 utterances. Medical queries were sourced from the r/AskDocs subreddit, while non-medical queries were pulled from random subreddits. Responses were generated from Amazon Alexa and DialoGPT-Large, and compared against high-rated human responses from Reddit. Annotations were collected from both an Advanced Nurse Practitioner (the gold standard) and Amazon Mechanical Turk workers to measure the "wisdom of the crowd."

Key Insights and Results

The Expertise Gap

Crowdworkers proved effective at identifying if a query was medical (Binary F1 of 0.86). However, they were significantly less reliable at grading the severity of the risk. Inter-annotator agreement between the expert and the aggregated crowd dropped from $\alpha = 0.86$ for binary tasks to $\alpha = 0.58$ for ordinal risk grading. This suggests that while non-experts can filter data, clinicians are essential for defining safety boundaries in high-stakes scenarios.

Classifier Blindness to "Critical" Cases

The authors fine-tuned BERT-base models for binary, ternary, and ordinal classification. While BERT achieved a high F1-score of 0.93 for binary medical query detection, its performance plummeted on risk grading (Macro F1 of 0.45 for queries).

Confusion matrices for BERT classifiers on query seriousness (left) and response risk (right), highlighting the failure to predict critical/high-risk labels.

As shown in the confusion matrices, the models tended to "regress to the mean," classifying life-threatening queries as merely "non-serious." Crucially, the model never predicted the "Critical" or "Very High Risk" labels, representing a catastrophic failure in the most dangerous cases.

System Safety Profiles

  • Alexa: Generally "safe" by default, often providing "No information" (Level 0) or stating "I don't know."
  • DialoGPT: As an end-to-end generative model, it was more prone to providing incoherent or high-risk advice, occasionally attempting to diagnose or treat serious conditions without a safety net.

Conclusion and Future Work

The primary challenge in training these models is data imbalance; truly "Critical" medical emergencies are rare in public datasets like Reddit. Future work must focus on data augmentation—perhaps through adversarial attacks or synthetic generation—to provide models with enough examples of high-stakes situations. Furthermore, safety thresholds should be dynamic, allowing systems to shift their behavior based on whether they are general-purpose assistants or specialized medical triage tools.

Summary updated

Abstract

Conversational AI systems can engage in unsafe behaviour when handling users' medical queries that can have severe consequences and could even lead to deaths. Systems therefore need to be capable of both recognising the seriousness of medical inputs and producing responses with appropriate levels of risk. We create a corpus of human written English language medical queries and the responses of different types of systems. We label these with both crowdsourced and expert annotations. While individual crowdworkers may be unreliable at grading the seriousness of the prompts, their aggregated labels tend to agree with professional opinion to a greater extent on identifying the medical queries and recognising the risk types posed by the responses. Results of classification experiments suggest that, while these tasks can be automated, caution should be exercised, as errors can potentially be very serious.
BibTeX
@inproceedings{abercrombie2022risk,
  author = {Gavin Abercrombie and Verena Rieser},
  title = {{Risk-graded Safety for Handling Medical Queries in Conversational AI}},
  booktitle = {AACL/IJCNLP},
  year = {2022},
  doi = {10.18653/v1/2022.aacl-short.30},
}
Figure