Project PRIMUS at EHRSQL 2024 : Text-to-SQL Generation using Large Language Model for EHR Analysis

Sourav Joy, Rohan Ahmed, Argha Saha, Minhaj Habil, Utsho Das, Partha BhowmikNAACL 2024

TL;DR

This paper explores using the sqlcoders large language model (LLM) for automatically translating natural language questions into SQL queries for Electronic Health Records (EHRs). The authors highlight the challenges of querying complex EHR databases and propose a prompt engineering approach to improve LLM performance in this domain, demonstrating its effectiveness on a healthcare-specific dataset.

What Problem Does This Paper Address?

Modern healthcare relies heavily on Electronic Health Records (EHRs) to store vast amounts of patient data. However, accessing and analyzing this data is often hindered by the complexity of traditional relational database querying. Existing methods, like pre-defined rule systems or manual SQL query writing, are either too rigid or require specialized expertise, creating a bottleneck for clinicians and researchers. This paper aims to bridge this gap by enabling users to query EHR databases using natural language, making data retrieval more accessible and streamlined.

Methods

The core of the proposed system is the sqlcoders model, a pre-trained LLM specifically designed for text-to-SQL translation. This model leverages transformer architectures with attention mechanisms, trained on pairs of natural language questions and their corresponding SQL queries. The authors emphasize the importance of "prompt engineering" – carefully crafting the input to the LLM. Their approach involves creating a comprehensive prompt that includes:

  • Task Description: Clearly stating the goal of generating an SQL query.
  • Question Placeholder: A designated spot for the natural language question.
  • Schema Description: Providing the database schema (tables, attributes, data types) to guide the model.
  • Instructions: Optional guidance for specific scenarios (e.g., handling missing data, performing calculations).
  • Answer Placeholder: Where the model outputs the generated SQL query or "null" for unanswerable questions.

This structured prompt helps the sqlcoders model understand the user's intent, the database structure, and the necessary SQL constructs to generate accurate queries.

Datasets

  • EHR Dataset: The MIMIC-IV dataset is used as the underlying data source. This dataset contains various tables and attributes relevant to patient information, such as diagnoses, medications, and patient IDs.
  • Question-SQL Pairs: The authors created a collection of question-SQL pairs specifically tailored to the healthcare domain, using the EHR data to generate these pairs.

Key Insights and Ideas

  • Natural Language as a Query Interface: The paper strongly advocates for natural language interfaces as a way to democratize access to complex data like EHRs, reducing the reliance on specialized SQL knowledge.
  • The Power of Prompt Engineering: The effectiveness of LLMs for text-to-SQL is significantly influenced by how the prompt is constructed. A well-designed prompt, incorporating schema information and task context, is crucial for guiding the model to generate accurate SQL.
  • Domain-Specific Challenges: Healthcare data presents unique challenges due to its complexity and the critical nature of accurate information retrieval. Fine-tuning LLMs for this specific domain is essential.
  • Evaluating Robustness: The paper introduces and utilizes different "Reliability Scoring" (RS) schemes to evaluate not just accuracy but also the model's robustness to errors, highlighting the importance of penalties for incorrect predictions.

Key Results

The model's performance was evaluated using four RS schemes:

  • RS(0) (No Penalty): The model achieved a score of 14.14, indicating it can generate some correct SQL queries. However, this score doesn't penalize errors, so it doesn't fully reflect true accuracy.
  • RS(5) (Moderate Penalty): The score dropped significantly to -349.61, suggesting a high number of incorrect predictions that are magnified by the penalty.
  • RS(10) (Main Evaluation Metric): The score further decreased to -713.37, emphasizing the model's shortcomings when stricter penalties are applied.
  • RS(N) (Strict Penalty): The score plummeted to -84885.86, indicating severe limitations. Even a single mistake can drastically impact the overall score, highlighting a lack of robustness.

The substantial drop in scores across the RS schemes demonstrates the model's sensitivity to errors and its inability to consistently generate reliable queries in the healthcare domain.

Limitations and Future Work

The paper acknowledges the model's inability to achieve positive scores under most RS scenarios, indicating a fundamental limitation in generating accurate SQL queries. The significant drop in score with increasing penalty severity highlights the model's susceptibility to errors and lack of robustness.

Future work suggested includes:

  • Exploring different model architectures.
  • Implementing enhanced error handling mechanisms.
  • Incorporating human expertise to further improve performance.

The authors believe these directions hold promise for significant advancements in text-to-SQL generation for EHR analysis.

Summary updated

BibTeX
@inproceedings{joy2024project,
  author = {Sourav Joy and Rohan Ahmed and Argha Saha and Minhaj Habil and Utsho Das and Partha Bhowmik},
  title = {{Project PRIMUS at EHRSQL 2024 : Text-to-SQL Generation using Large Language Model for EHR Analysis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure