Saama Technologies at EHRSQL 2024: SQL Generation through Classification Answer Selector by LLM

Mohammed Jabir, Kamal Kanakarajan, Malaikannan SankarasubbuNAACL 2024

TL;DR

This paper introduces SCAS, a novel framework for generating SQL queries from natural language questions in Electronic Health Records (EHR) databases. SCAS uses a two-module approach: a Classification Answer Selector (CAS) to determine if a question is answerable, and a SQL Generation (SG) module to create the SQL query for answerable questions. The system achieved a respectable 7th place in the EHRSQL 2024 challenge, demonstrating the effectiveness of LLMs in this complex domain.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) are a treasure trove of patient data, crucial for clinical decision-making. However, accessing this information often requires complex SQL queries, which can be a bottleneck for healthcare professionals. The EHRSQL task aims to bridge this gap by developing systems that can automatically translate natural language questions into executable SQL queries, while also identifying questions that cannot be answered from the database. This is particularly challenging due to the complexity and sensitivity of EHR data, requiring high accuracy and reliability.

Methods

The core of the SCAS framework lies in its two-stage approach:

  • Classification Answer Selector (CAS) Module: This module determines whether a given question can be answered by the EHR database. It employs a two-pronged strategy:
  • Classical Machine Learning: This involves using traditional models like MultinomialNB, SGD, and CatBoost with feature selection (TF-IDF) and ensemble methods. Predictions are made based on probabilities, with a threshold of 0.4 to classify questions.
  • Fine-tuned LLM: A pre-trained LLM (CodeLlama-7b-Instruct-hf) is fine-tuned for classification.
  • Selector: An advanced LLM (Azure's OpenAI GPT-3.5-turbo) is used to combine the outputs of the classical and LLM classifiers to make the final answerability decision. This leverages in-context learning for improved accuracy.
  • SQL Generation (SG) Module: For questions identified as answerable by the CAS module, this module generates the corresponding SQL query. It utilizes the same fine-tuned CodeLlama LLM used in the CAS module, but specifically for instruction tuning on generating SQL.

The entire system is fine-tuned using efficient parameter tuning methods like LoRA and the AdamW optimizer with a cosine learning rate scheduler.

Datasets

The research utilizes the EHRSQL 2024 dataset, which is derived from the MIMIC-IV open-access database. This dataset includes both answerable and unanswerable questions related to EHR data. The dataset comprises 5124 training, 1163 validation, and 1162 testing samples.

Key Insights and Ideas

  • Two-stage approach is effective: Separating the task into answerability classification and SQL generation proves beneficial, allowing for more focused optimization of each stage.
  • LLMs excel at both classification and generation: The study demonstrates that LLMs, particularly when fine-tuned, can perform well on both classifying question answerability and generating complex SQL queries.
  • Combining classical and LLM classifiers enhances robustness: The CAS module's use of both traditional ML models and a fine-tuned LLM, with an LLM selector, likely contributes to its accuracy in determining answerability.
  • In-context learning is powerful: The paper highlights the utility of in-context learning, especially for the selector module, to leverage LLMs' capabilities without extensive fine-tuning for every specific task.
  • Fine-tuning is crucial for performance: While in-context learning is valuable, the results suggest that fine-tuning LLMs on task-specific data (like EHR SQL generation) leads to significant performance gains.

Key Results

The SCAS system achieved a 7th position on the EHRSQL 2024 leaderboard with a Reliability Score (RS) of 53.21 on the official test set. During the development phase, the system demonstrated strong performance, with the SQL generator module achieving an impressive RS(0) score of 77.03. The paper also notes that their fine-tuned CodeLlama model outperformed a GPT-3.5-turbo baseline, even with fewer parameters.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Focus on Codellama: The research primarily focused on Codellama models, and it's noted that other fine-tuned models like SQLCoder did not perform as well.
  • Limited unanswerable questions: The EHRSQL dataset has a relatively small number of unanswerable questions (450 out of 5124), which might limit the training of robust unanswerable question detection.
  • False negatives in CAS: An error analysis revealed a higher rate of false negatives in the CAS module, meaning some answerable questions were incorrectly classified as unanswerable. This impacts the overall pipeline as only answerable questions proceed to SQL generation.

Future work could involve:

  • Generating synthetic data: Creating synthetic data for unanswerable questions using models like GPT-4 could improve the system's ability to identify them.
  • Improving unanswerable question identification: Further research is needed to enhance the system's capability in identifying which user queries can be successfully converted to SQL, thereby improving overall reliability.

Summary updated

BibTeX
@inproceedings{jabir2024saama,
  author = {Mohammed Jabir and Kamal Kanakarajan and Malaikannan Sankarasubbu},
  title = {{Saama Technologies at EHRSQL 2024: SQL Generation through Classification Answer Selector by LLM}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure