TL;DR
This paper presents the first user study with real healthcare professionals comparing single-turn and conversational (multi-turn) text-to-SQL interfaces for accessing electronic health records (EHR). The study reveals that conversational interaction is a practical necessity—not just a convenience—because nearly 25% of naturally formulated clinical queries require clarification to resolve domain-specific ambiguities.
What Problem Does This Paper Address?
Electronic Health Records (EHRs) contain vast amounts of data critical for research and decision support, but this data is locked behind complex relational schemas. For a physician to extract a patient cohort, they typically need high proficiency in SQL and deep knowledge of the underlying database structure—a significant barrier to entry.
While Large Language Models (LLMs) have shown promise in "text-to-SQL" (translating natural language to code), most evaluations rely on static benchmarks like Spider or EHRSQL. These benchmarks use "fully specified" questions where every detail is pre-defined. In reality, clinical questions are often messy and ambiguous (e.g., "Show me patients with high blood pressure"—does that mean a single reading, an average, or a formal diagnosis code?). This paper investigates whether a conversational interface, which allows for iterative clarification, actually improves usability and trust for doctors in the field.
Methods
The researchers conducted a within-subjects user study with 14 physicians (plus 7 additional professionals in pilot studies). The workflow, illustrated in Fig. 1: Overview of the user-study design, consisted of:
- Phase 1 (Controlled Comparison): Participants used both a single-turn and a multi-turn interface to answer a set of 45 predefined clinical queries.
- Phase 2 (Ecological Validity): Participants freely explored the database using their own clinical or research questions to simulate real-world information-seeking behavior.
- Post-Study Assessment: Participants completed standardized surveys, including the System Usability Scale (SUS) and the Unified Theory of Acceptance and Use of Technology (UTAUT).
The underlying engine for both interfaces was GPT-5.4, which was selected after pilot testing showed it outperformed earlier versions in accuracy and latency.
Datasets
The study utilized the MIMIC-IV Clinical Database Demo, a publicly available, de-identified subset of real-world EHR data containing 100 ICU patients across 31 tables. This dataset includes structured data like admissions, laboratory results, and medication records.
Key Insights and Ideas
- Ambiguity is the Clinical Norm: While predefined benchmark questions almost never triggered clarification (1.8%), freely composed queries by physicians triggered clarification requests in 24.8% of cases.
- Clarification is Domain-Specific: It wasn't just "bad grammar" causing issues. As shown in Fig. 4: Distribution of clarification triggers, 64.3% of clarification episodes stemmed from EHR or clinical-domain knowledge, such as defining a "cohort" or choosing between different data sources for the same measurement.
- Trust is Interactive: Participants trusted the conversational interface more (71%) not necessarily because it was more accurate on the first try, but because they could interrogate, verify, and refine the results. The ability to "double-check" through dialogue was a major driver of confidence.
- The "SQL Barrier" is Real: Despite being frequent EHR users, most participants had very low SQL proficiency, as seen in Fig. 2: Participant background characteristics, highlighting the massive potential for natural-language interfaces in healthcare.
Key Results
The conversational multi-turn interface was the clear winner across all metrics:
- Usability: The mean SUS score jumped from 68.2 (industry average) for the single-turn interface to 85.9 ("Best Imaginable") for the conversational interface ($p < 0.001$).
- Technology Acceptance: Significant advantages were found in performance expectancy, effort expectancy, and behavioral intention. These results are detailed in Fig. 3: System usability and technology acceptance.
- Preference: 93% of participants reported greater overall satisfaction with the conversational interface, and 86% preferred it for real-world adoption. Notably, zero participants preferred or trusted the single-turn interface more.
Limitations and Future Work
The authors acknowledge several limitations:
- Sample Size and Diversity: The study involved 14 physicians at a single center. While the results were statistically significant, broader studies including nurses and data scientists are needed.
- Dataset Scale: The MIMIC-IV Demo is small (100 patients). Real-world hospital databases are orders of magnitude larger and more complex, which might increase the frequency of ambiguity and the need for even more robust clarification.
- Error Propagation: In multi-turn dialogues, an early mistake by the LLM (e.g., an incorrect cohort definition) can propagate through the entire conversation. Improving "context maintenance" is a critical area for future engineering.
- Accuracy Gap: Human labeling showed the model's overall accuracy was around 72%. While high, the authors note this still leaves "considerable room for improvement" before these tools can be deployed for high-stakes clinical decision-making.
Summary updated