Don't be my Doctor! Recognizing Healthcare Advice in Large Language Models

Pengyuan Li, Kellen Cheng, Anna Lisa Gentile, Pengyuan Li 0001, Chad DeLuca, Guang-Jie RenEMNLP 2024 · DOI

Abstract

As Large Language Models (LLMs) become ubiquitous as virtual assistants, users increasingly turn to them for health queries. This creates a massive liability: an LLM providing direct medical advice can lead to user harm or legal malpractice. This paper introduces HeAL (HEalth Advice in LLMs), a new benchmark designed to help AI guardrails distinguish between general health information and actionable medical advice. By testing both massive LLMs and smaller fine-tuned models, the study reveals that while top-tier models like GPT-4o are highly cautious, they still struggle with the subtle nuances of personal anecdotes and conversational health tips.

The Challenge of Medical Guardrails

The core challenge in automating medical safety is nuance. There is a thin line between providing information (e.g., "Studies show Vitamin C supports immunity") and actionable advice (e.g., "You should take 1000mg of Vitamin C daily"). Existing safety benchmarks often rely on formal, academic medical texts, which do not reflect the messy, conversational way people actually talk on the internet.

The authors address this gap by creating a dataset that captures real-world, colloquial health interactions. This is critical because a model that is too permissive (False Negatives or "Escapes") risks giving dangerous advice, while a model that is too restrictive (False Positives or "Overkill") becomes unhelpful to the user.

Methodology

The researchers employed a multi-pronged approach to evaluate health-advice detection:

  1. Focused Crawling: Instead of broad web-scraping, they targeted specific health hubs (WebMD, Mayo Clinic, Everyday Health) and social media (Reddit). They used TF-IDF analysis to identify "seed keywords" (like the word "should") that are highly correlated with advice-giving.
  2. Model Evaluation: They compared two distinct strategies:
  3. Zero-Shot LLMs: Testing heavyweights like GPT-4o, LLaMA-3-70B, and Mixtral-8x7B without specific training on the task.
  4. Fine-tuned Classifiers: They took smaller, "encoder-only" models (BERT and RoBERTa) and fine-tuned them on a massive aggregate of 43,000 samples from various advice-related datasets.
  5. Error Analysis: The team conducted a case study on "Failure Modes" to see exactly where models trip up, specifically looking at False Positives (Overkill) and False Negatives (Escape).

The HeAL Dataset

The primary contribution is the HeAL benchmark, a "gold standard" set of 402 English samples, manually curated and annotated. It includes a mix of formal medical sites and informal Reddit threads to capture the diversity of online health discourse.

Examples of Non-Healthcare Advice (NHA) and Healthcare Advice (HA) from the HeAL dataset.

To train the BERT-based models, the authors combined five existing datasets (NeedAdvice, AskParents, SemEval 2019 Task 9, HealthE, and Health-Detection), resulting in a training set of approximately 43,812 samples.

Key Insights

  • The "Caution" Bias in GPT-4o: GPT-4o achieved the highest recall and the lowest "escape rate" (only 3.98%). However, it also had a high "overkill rate," frequently flagging harmless information as medical advice. This suggests GPT-4o is tuned to be "better safe than sorry."
  • The Conversational Gap: Models struggle significantly with personal anecdotes. If a user says, "In my experience, this bandage worked for me," humans might not see it as advice, but models often flag it as a suggestion.
  • Small Models Punching Up: BERT-Large, despite being orders of magnitude smaller than the LLMs, proved remarkably competitive when fine-tuned. It showed more consistent behavior without the extreme "overkill" tendencies of GPT-4o.
  • Imperatives are the Giveaway: The worst-performing model (BERT-Base) struggled most with "imperatives"—sentences that directly tell a user what to do. This highlights that even basic advice detection requires a sophisticated understanding of sentence structure and intent.

Key Results

The evaluation across various metrics shows that while LLMs are currently the best at catching advice, there is still a significant "Error Rate" (18-31%) across all models, indicating that medical guardrails are far from solved.

Performance comparison of various models on the HeAL benchmark.

Limitations and Future Work

While high-quality, the HeAL benchmark is relatively small (402 samples), and the authors acknowledge that a larger gold-standard set would be beneficial. Furthermore, the study focused largely on explicit advice; detecting implicit advice—where the suggestion is buried in subtext—remains an open challenge. Future work should focus on better algorithms to boost the performance of smaller BERT-style classifiers, which are cheaper and faster to deploy as real-time guardrails than massive LLMs.

Summary updated

BibTeX
@inproceedings{li2024don,
  author = {Pengyuan Li and Kellen Cheng and Anna Lisa Gentile and Pengyuan Li and Chad DeLuca and Guang-Jie Ren},
  title = {{Don't be my Doctor! Recognizing Healthcare Advice in Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-industry.72},
}
Figure