“What’s Up, Doc?”: Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasets

Akshay Paruchuri, Maryam Aziz, Rohit Vartak, Ayman Ali, Best Uchehara, Xin Liu, Ishan Chatterjee, Monica Agrawal

Abstract

Researchers curated HealthChat-11K, a first-of-its-kind dataset of 11,000 real-world healthcare conversations between users and Large Language Models (LLMs). By applying a new clinician-driven taxonomy, they revealed that users often engage in repetitive information-seeking loops, provide incomplete context, and frequently ask "leading questions" that can trick models into providing biased or inappropriate medical advice. This work highlights the critical gap between LLM performance on medical benchmarks and their safety in real-world conversational health assistance.

Introduction

While Large Language Models (LLMs) show impressive proficiency on medical exams and clinician-oriented benchmarks, these evaluations do not reflect how the general public actually uses them. Real-world health queries are often messy, multi-turn, and emotionally charged. Existing public datasets are typically limited to single-turn queries or synthetic interactions, leaving a gap in our understanding of patient-AI communication patterns. Without understanding natural pitfalls like sycophancy (the model's tendency to agree with the user) and incomplete context, we cannot safely deploy LLMs as conversational health assistants.

Methodology

The authors developed a multi-stage pipeline to transform raw interaction logs into a structured research asset, as illustrated in the research overview below:

Research overview: Taxonomy development, dataset curation, and empirical analysis.
  1. Dataset Curation: Millions of conversations from the LMSYS-Chat-1M and WildChat-1M datasets were filtered. Using Gemini 1.5 Pro with few-shot prompts, the team removed non-health, academic, fictional, and toxic content to isolate 11,000 genuine health-seeking dialogues.
  2. Clinician-Driven Taxonomy: In collaboration with clinicians, a hierarchical taxonomy was built. It classifies conversations into 21 specialties (e.g., Cardiology, Mental Health) and labels individual messages across four main categories: Providing Context (A), Seeking Information/Advice (B), Interaction Management (C), and Out-of-Scope (D).
  3. Automated Annotation: An LLM was used to apply these labels at scale, achieving a high concordance (0.78 F1-score) with human expert annotators.

The HealthChat-11K Dataset

The primary contribution of this work is HealthChat-11K, which contains 11,000 conversations and 25,000 user messages. Derived from real-world interaction logs, the dataset shows that while 56% of conversations are single-turn, a significant 16% extend to four or more turns, reflecting the complex nature of health inquiries.

Empirical Analysis and Results

The analysis revealed several critical patterns in how users interact with AI for health advice:

  • Information-Seeking Loops: Users are four times more likely to seek information (Category B) than provide context. There is a 57% chance that an information-seeking message is immediately followed by another, suggesting users get stuck in repetitive loops rather than balanced dialogues.
  • Specialty-Specific Behavior: In Mental Health, users typically start by providing context (describing symptoms). Conversely, in General Health, they often jump straight to questions without background.
  • The "Leading Question" Trap: Approximately 23% of treatment inquiries were "Leading Questions about a Specific Treatment" (LQSTs). For example, a user might ask "Is Drug Y best for X?" rather than asking for the best treatment. This is dangerous because LLMs often exhibit sycophancy, validating inappropriate treatments suggested by the user.
  • Data Distributions: The most common specialties were General Health, Mental Health, and Diet/Nutrition. The most frequent message types were "Treatment Inquiry" (B5) and "Disease Information Inquiry" (B8).
Distributions of medical specialties and message-level taxonomy codes in HealthChat-11K.

Discussion and Future Work

The study identified a recurring pattern of Incomplete Context, where users ask for a diagnosis and only later reveal critical medical history that could change the advice entirely. While explicit emotional affect was rare (~3%), these moments often signaled the end of a conversation, marking them as critical points for AI "conversational repair."

Future work should address the English-only limitation of this dataset and shift focus to analyzing LLM responses to these tricky user behaviors. Additionally, some users were found to be impersonating clinicians to "jailbreak" model guardrails, a behavior requiring more robust detection. The team has released their code and artifacts on GitHub to encourage the development of safer healthcare benchmarks.

Summary updated

Abstract

People are increasingly seeking healthcare information from large language models (LLMs) via interactive chatbots, yet the nature and inherent risks of these conversations remain largely unexplored.In this paper, we filter large-scale conversational AI datasets to achieve HealthChat-11K, a curated dataset of 11K real-world conversations composed of 25K user messages.We use HealthChat-11K and a clinician-driven taxonomy for how users interact with LLMs when seeking healthcare information in order to systematically study user interactions across 21 distinct health specialties.Our analysis reveals insights into the nature of how and why users seek health information, such as common interactions, instances of incomplete context, affective behaviors, and interactions (e.g., leading questions) that can induce sycophancy, underscoring the need for improvements in the healthcare support capabilities of LLMs deployed as conversational AI.Code and artifacts to retrieve our analyses and combine them into a curated dataset can be found here: https: //github.com/yahskapar/HealthChatWhy do I find myself tensing my muscles all the time?So tensing muscles is a way of numbing yourself?I want you to explain it in the context of psychology
BibTeX
@misc{paruchuri2025whats,
  author = {Akshay Paruchuri and Maryam Aziz and Rohit Vartak and Ayman Ali and Best Uchehara and Xin Liu and Ishan Chatterjee and Monica Agrawal},
  title = {{“What’s Up, Doc?”: Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasets}},
  year = {2025},
  eprint = {2506.21532},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.21532},
}
Figure