Abstract
Electronic Health Records (EHRs) are critical repositories of patient medical histories, yet their relational structure often requires SQL expertise to navigate. The EHRSQL 2024 shared task was established to promote the development of question-answering systems that can reliably translate natural language into SQL. A key focus of the task is "reliability"—the ability of a model to abstain from answering when a query is unanswerable or ambiguous, rather than providing incorrect information. This paper summarizes the task objectives, the EHRSQL dataset, the methodologies employed by participating teams, and the current state of reliable text-to-SQL modeling in the clinical domain.
1. Introduction
In the clinical environment, the ability to quickly retrieve patient data from Electronic Health Records (EHRs) is vital for informed decision-making. However, EHRs are typically stored in complex relational databases, creating a technical barrier for healthcare professionals who may lack proficiency in query languages like SQL. Text-to-SQL models, powered by Large Language Models (LLMs), offer a promising solution by translating natural language questions into executable queries.
Despite their potential, standard Text-to-SQL models often suffer from "overconfidence," attempting to generate answers even when questions are ambiguous, refer to missing data, or fall outside the system's capabilities. In healthcare, a hallucinated or incorrect answer—such as an erroneous medication dosage or a misattributed lab result—poses significant safety risks. The EHRSQL 2024 shared task was designed to address this by prioritizing reliability: the capacity of a system to provide correct answers or gracefully abstain.
2. The EHRSQL Dataset and Task Design
The shared task utilized a specialized version of the EHRSQL dataset, built upon the MIMIC-IV demo database, which contains de-identified data for 100 patients. The dataset was curated to reflect real-world clinical inquiries and included two primary categories of questions:
- Answerable Questions: These were derived from real-world questions posed by healthcare professionals and mapped to SQL templates.
- Unanswerable Questions: To test reliability, the dataset included "naturally" unanswerable questions (e.g., requests for data types not present in the schema) and adversarial examples from the TrustSQL dataset (e.g., queries referencing non-existent columns or tables).
A critical component of the 2024 task was the inclusion of "unseen" question templates in the test set. This forced models to demonstrate generalization capabilities rather than relying on memorized patterns from the training data.
3. Methodologies
Participating teams generally adopted one of two architectural strategies:
3.1 Unified Approaches
The most successful systems utilized a unified LLM approach. In these models, a single powerful agent (such as GPT-4 or a fine-tuned variant) is responsible for both SQL generation and determining "answerability." The top-performing team, a collaboration between LG AI Research and KAIST, employed self-training with pseudo-labeling. This technique helped the model learn the boundaries of its own knowledge, improving its ability to identify unanswerable queries.
3.2 Pipeline-Based Approaches
Other teams utilized multi-stage pipelines. These systems typically involve an answerability classifier to filter out unanswerable questions, a SQL generator, and sometimes an error detector to validate the logic of the generated query. While modular, these pipelines often struggled to match the performance of unified models, likely because unified models can leverage broader internal reasoning to contextualize the relationship between the question and the database schema.
4. Evaluation: The Reliability Score
The primary metric for the task was the Reliability Score (RS). Unlike standard accuracy, RS heavily penalizes incorrect predictions. In the RS(10) configuration, the penalty for a single incorrect answer is ten times the reward for a correct one. This scoring system incentivizes models to abstain unless they have high confidence.
The task also featured an RS(N) metric—the most rigorous evaluation where any single mistake results in a negative overall score, effectively demanding "perfect" reliability.
5. Results and Key Findings
The results of the shared task highlighted both the progress in the field and the remaining challenges:
- Top Performance: The winning team achieved an RS(10) score of 81.32. This was accomplished through a combination of unified prompting and domain-specific self-training.
- Baseline Comparison: An "Abstain-All" baseline, which provides no answers, would achieve a score of 20.0 (reflecting the 20% of the test set that was unanswerable). While most teams exceeded this, the high penalty for errors led one team to a score of -713.37, illustrating the volatility of overconfident models.
- The Reliability Gap: No participating team achieved a positive score on the RS(N) metric. This indicates that while current models are becoming more reliable, they are not yet "clinical-grade" in terms of absolute error prevention.
6. Conclusion and Future Directions
The EHRSQL 2024 shared task demonstrates that while LLMs are increasingly capable of navigating complex medical schemas, achieving the level of reliability required for clinical deployment remains an open challenge. Future research must address the "generalization bottleneck" where models struggle with unseen SQL structures. Furthermore, expanding the scope from structured tables to multimodal data—including clinical notes and imaging—will be essential for creating a truly holistic and reliable question-answering system for healthcare.
Summary updated