Abstract
Researchers curated HealthChat-11K, a first-of-its-kind dataset of 11,000 real-world healthcare conversations between users and Large Language Models (LLMs). By applying a new clinician-driven taxonomy, they revealed that users often engage in repetitive information-seeking loops, provide incomplete context, and frequently ask "leading questions" that can trick models into providing biased or inappropriate medical advice. This work highlights the critical gap between LLM performance on medical benchmarks and their safety in real-world conversational health assistance.
Introduction
While Large Language Models (LLMs) show impressive proficiency on medical exams and clinician-oriented benchmarks, these evaluations do not reflect how the general public actually uses them. Real-world health queries are often messy, multi-turn, and emotionally charged. Existing public datasets are typically limited to single-turn queries or synthetic interactions, leaving a gap in our understanding of patient-AI communication patterns. Without understanding natural pitfalls like sycophancy (the model's tendency to agree with the user) and incomplete context, we cannot safely deploy LLMs as conversational health assistants.
Methodology
The authors developed a multi-stage pipeline to transform raw interaction logs into a structured research asset, as illustrated in the research overview below:
- Dataset Curation: Millions of conversations from the LMSYS-Chat-1M and WildChat-1M datasets were filtered. Using Gemini 1.5 Pro with few-shot prompts, the team removed non-health, academic, fictional, and toxic content to isolate 11,000 genuine health-seeking dialogues.
- Clinician-Driven Taxonomy: In collaboration with clinicians, a hierarchical taxonomy was built. It classifies conversations into 21 specialties (e.g., Cardiology, Mental Health) and labels individual messages across four main categories: Providing Context (A), Seeking Information/Advice (B), Interaction Management (C), and Out-of-Scope (D).
- Automated Annotation: An LLM was used to apply these labels at scale, achieving a high concordance (0.78 F1-score) with human expert annotators.
The HealthChat-11K Dataset
The primary contribution of this work is HealthChat-11K, which contains 11,000 conversations and 25,000 user messages. Derived from real-world interaction logs, the dataset shows that while 56% of conversations are single-turn, a significant 16% extend to four or more turns, reflecting the complex nature of health inquiries.
Empirical Analysis and Results
The analysis revealed several critical patterns in how users interact with AI for health advice:
- Information-Seeking Loops: Users are four times more likely to seek information (Category B) than provide context. There is a 57% chance that an information-seeking message is immediately followed by another, suggesting users get stuck in repetitive loops rather than balanced dialogues.
- Specialty-Specific Behavior: In Mental Health, users typically start by providing context (describing symptoms). Conversely, in General Health, they often jump straight to questions without background.
- The "Leading Question" Trap: Approximately 23% of treatment inquiries were "Leading Questions about a Specific Treatment" (LQSTs). For example, a user might ask "Is Drug Y best for X?" rather than asking for the best treatment. This is dangerous because LLMs often exhibit sycophancy, validating inappropriate treatments suggested by the user.
- Data Distributions: The most common specialties were General Health, Mental Health, and Diet/Nutrition. The most frequent message types were "Treatment Inquiry" (B5) and "Disease Information Inquiry" (B8).
Discussion and Future Work
The study identified a recurring pattern of Incomplete Context, where users ask for a diagnosis and only later reveal critical medical history that could change the advice entirely. While explicit emotional affect was rare (~3%), these moments often signaled the end of a conversation, marking them as critical points for AI "conversational repair."
Future work should address the English-only limitation of this dataset and shift focus to analyzing LLM responses to these tricky user behaviors. Additionally, some users were found to be impersonating clinicians to "jailbreak" model guardrails, a behavior requiring more robust detection. The team has released their code and artifacts on GitHub to encourage the development of safer healthcare benchmarks.
Summary updated