Conversational Interaction for LLM-Based Clinical Text-to-SQL: A User Study with Healthcare Professionals

Jun-Min Lee, Suyoung Yoo, Gyubok Lee, Edward Choi, Meong Hi SonarXiv 2026 · arXiv

TL;DR

This paper presents the first user study with real healthcare professionals comparing single-turn and conversational (multi-turn) text-to-SQL interfaces for accessing electronic health records (EHR). The study reveals that conversational interaction is a practical necessity—not just a convenience—because nearly 25% of naturally formulated clinical queries require clarification to resolve domain-specific ambiguities.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) contain vast amounts of data critical for research and decision support, but this data is locked behind complex relational schemas. For a physician to extract a patient cohort, they typically need high proficiency in SQL and deep knowledge of the underlying database structure—a significant barrier to entry.

While Large Language Models (LLMs) have shown promise in "text-to-SQL" (translating natural language to code), most evaluations rely on static benchmarks like Spider or EHRSQL. These benchmarks use "fully specified" questions where every detail is pre-defined. In reality, clinical questions are often messy and ambiguous (e.g., "Show me patients with high blood pressure"—does that mean a single reading, an average, or a formal diagnosis code?). This paper investigates whether a conversational interface, which allows for iterative clarification, actually improves usability and trust for doctors in the field.

Methods

The researchers conducted a within-subjects user study with 14 physicians (plus 7 additional professionals in pilot studies). The workflow, illustrated in Fig. 1: Overview of the user-study design, consisted of:

Fig. 1: Overview of the user-study design. Participants completed a within-subject comparison of single-turn and conversational multi-turn text-to-SQL interfaces, con- sisting of a controlled comparison on pre-defined questions (Phase 1) followed by open-ended querying (Phase 2).
  1. Phase 1 (Controlled Comparison): Participants used both a single-turn and a multi-turn interface to answer a set of 45 predefined clinical queries.
  2. Phase 2 (Ecological Validity): Participants freely explored the database using their own clinical or research questions to simulate real-world information-seeking behavior.
  3. Post-Study Assessment: Participants completed standardized surveys, including the System Usability Scale (SUS) and the Unified Theory of Acceptance and Use of Technology (UTAUT).

The underlying engine for both interfaces was GPT-5.4, which was selected after pilot testing showed it outperformed earlier versions in accuracy and latency.

Datasets

The study utilized the MIMIC-IV Clinical Database Demo, a publicly available, de-identified subset of real-world EHR data containing 100 ICU patients across 31 tables. This dataset includes structured data like admissions, laboratory results, and medication records.

Key Insights and Ideas

  • Ambiguity is the Clinical Norm: While predefined benchmark questions almost never triggered clarification (1.8%), freely composed queries by physicians triggered clarification requests in 24.8% of cases.
  • Clarification is Domain-Specific: It wasn't just "bad grammar" causing issues. As shown in Fig. 4: Distribution of clarification triggers, 64.3% of clarification episodes stemmed from EHR or clinical-domain knowledge, such as defining a "cohort" or choosing between different data sources for the same measurement.
  • Trust is Interactive: Participants trusted the conversational interface more (71%) not necessarily because it was more accurate on the first try, but because they could interrogate, verify, and refine the results. The ability to "double-check" through dialogue was a major driver of confidence.
  • The "SQL Barrier" is Real: Despite being frequent EHR users, most participants had very low SQL proficiency, as seen in Fig. 2: Participant background characteristics, highlighting the massive potential for natural-language interfaces in healthcare.
Fig. 4: Distribution of clarification triggers
Fig. 2: Distribution of participant background characteristics (n = 14), including EHR (or EMR)/CDW usage frequency, SQL proficiency, statistical software profi- ciency, and conversational AI usage frequency.

Key Results

The conversational multi-turn interface was the clear winner across all metrics:

  • Usability: The mean SUS score jumped from 68.2 (industry average) for the single-turn interface to 85.9 ("Best Imaginable") for the conversational interface ($p < 0.001$).
  • Technology Acceptance: Significant advantages were found in performance expectancy, effort expectancy, and behavioral intention. These results are detailed in Fig. 3: System usability and technology acceptance.
  • Preference: 93% of participants reported greater overall satisfaction with the conversational interface, and 86% preferred it for real-world adoption. Notably, zero participants preferred or trusted the single-turn interface more.
Fig. 3: System usability, technology acceptance, and interface preference for the single- turn versus conversational multi-turn interfaces (n = 14). (a) SUS scores (0–100); the dashed line indicates the industry-average score of approximately 68. (b) UTAUT construct scores (1–5): Performance Expectancy (PE), Effort Expectancy (EE), Self- Efficacy (SE), Anxiety (ANX), Behavioral Intention (BI), and the average of the four positively oriented constructs, Avg. UTAUT (P). Bars represent the mean and error bars indicate the standard deviation. Statistical significance is based on the primary paired test for each metric (Section 4.5.2): ∗p < 0.05, ∗∗p < 0.01, ∗∗∗p < 0.001; n.s., not significant. Higher ANX scores indicate greater anxiety. (c) Distribution of participant responses to the three post-study forced-choice comparison questions (n = 14): overall satisfaction, preferred interface for real-world adoption, and trust in the returned results.

Limitations and Future Work

The authors acknowledge several limitations:

  • Sample Size and Diversity: The study involved 14 physicians at a single center. While the results were statistically significant, broader studies including nurses and data scientists are needed.
  • Dataset Scale: The MIMIC-IV Demo is small (100 patients). Real-world hospital databases are orders of magnitude larger and more complex, which might increase the frequency of ambiguity and the need for even more robust clarification.
  • Error Propagation: In multi-turn dialogues, an early mistake by the LLM (e.g., an incorrect cohort definition) can propagate through the entire conversation. Improving "context maintenance" is a critical area for future engineering.
  • Accuracy Gap: Human labeling showed the model's overall accuracy was around 72%. While high, the authors note this still leaves "considerable room for improvement" before these tools can be deployed for high-stakes clinical decision-making.

Summary updated

Abstract

Conversational text-to-SQL interfaces could give healthcare professionals natural-language access to electronic health record (EHR) data, but evidence for their real-world usability remains limited: prior evaluations have relied largely on curated benchmarks with fully specified questions or on simulated users. We report a user study in which 14 physicians who regularly work with EHR data used both a single-turn and a conversational multi-turn text-to-SQL interface over the MIMIC-IV database in a within-subjects design; dialogue analyses additionally drew on interaction logs from 21 healthcare professionals across the main study and two pilot studies. The conversational interface was consistently favored: mean System Usability Scale scores rose from 68.2, near the industry average, to 85.9 (p<0.001), with significant advantages across technology-acceptance constructs including performance expectancy, effort expectancy, self-efficacy, and behavioral intention. In post-study forced-choice comparisons, 93% of participants reported greater overall satisfaction with the conversational interface, 86% preferred it for real-world adoption, and 71% trusted its results more; no participant preferred or trusted the single-turn interface more. Turn-level analysis of 425 logged user turns showed that freely composed queries triggered clarification requests in 24.8% of cases versus 1.8% for predefined, fully specified questions, and 64.3% of clarification episodes stemmed from EHR- or clinical-domain knowledge. Conversational interaction is a practical prerequisite, rather than a convenience, for natural-language EHR access. The frequent need for clarification in realistic clinical queries further underscores the importance of conversational interfaces, while benchmark-style evaluation can substantially underrepresent the ambiguity of real clinical use.
BibTeX
@misc{lee2026conversational,
  author = {Jun-Min Lee and Suyoung Yoo and Gyubok Lee and Edward Choi and Meong Hi Son},
  title = {{Conversational Interaction for LLM-Based Clinical Text-to-SQL: A User Study with Healthcare Professionals}},
  year = {2026},
  eprint = {2611.00605},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00605},
}
Figure