ProbGate at EHRSQL 2024: Enhancing SQL Query Generation Accuracy through Probabilistic Threshold Filtering and Error Handling

Sangryul Kim, Donghee Han, Sehyun KimNAACL 2024

TL;DR

This paper introduces ProbGate, a novel method for improving the accuracy of SQL query generation from natural language, particularly for sensitive domains like healthcare. ProbGate effectively filters out unanswerable queries by leveraging probabilistic thresholds and SQL execution error handling, achieving state-of-the-art results on the EHRSQL 2024 benchmark.

What Problem Does This Paper Address?

The ability to translate natural language questions into SQL queries (Text2SQL) has immense potential, especially in domains like healthcare where accessing patient records is crucial. However, a significant challenge arises from "unanswerable" queries – those that cannot be answered from the database due to missing information or incompatibility with the schema. Existing methods often struggle to reliably distinguish these unanswerable queries, leading to potentially misleading results if not carefully inspected. This is particularly problematic in healthcare, where incorrect information can have severe consequences. Furthermore, many advanced Text2SQL models are accessed via APIs, limiting direct access to their internal parameters for fine-grained analysis and filtering. The paper aims to address this by proposing a robust and adaptable filtering mechanism that can be applied even without direct model access.

Methods

The core of the proposed solution, ProbGate, is a two-stage filtering process applied after an initial fine-tuning of a large language model (LLM), specifically GPT-3.5-turbo.

  1. Probabilistic Threshold Filtering (ProbGate): This stage leverages the inherent uncertainty in LLM-generated text. The authors hypothesize that tokens with low log probabilities in a generated SQL query are indicative of potential "hallucinations" or unanswerable content. ProbGate calculates the log probabilities of tokens within generated SQL queries, excluding common SQL reserved words to focus on more informative parts. It then identifies a set of "unanswerable" candidates based on a probabilistic threshold derived from the distribution of these log probabilities. The paper suggests that fine-tuning the LLM on answerable queries helps create a clearer distinction in log probability distributions between answerable and unanswerable queries.
  2. Grammatical Errors Filtering (GEF): After ProbGate filters out likely unanswerable queries, the remaining "answerable" candidates are executed against the actual database. If a query fails to execute due to grammatical or schema errors, it is then classified as unanswerable. This step is crucial because even if a query is semantically plausible, minor syntax errors can render it invalid. The authors argue that treating such cases as unanswerable is safer than returning incorrect results, especially in high-stakes domains.

The overall pipeline involves fine-tuning GPT-3.5-turbo on a dataset of answerable Text2SQL pairs, then applying ProbGate to filter unanswerable queries from the test set, followed by GEF.

Datasets

The primary dataset used in this study is the EHRSQL Shared Task dataset (Lee et al., 2022, 2024b). This dataset is specifically designed for Text2SQL tasks within the Electronic Health Records (EHR) domain. It includes natural language questions and their corresponding SQL queries, with a crucial distinction: it contains both answerable and unanswerable questions. The EHRSQL-2024 task dataset comprises training data (5124 examples) and test data (1167 examples). The training data was used to fine-tune the LLM, with unanswerable examples being excluded during this initial fine-tuning phase. The test dataset contains questions, including types of unanswerable questions not present in the training data, making it a challenging evaluation set.

Key Insights and Ideas

  • Log Probability as a Confidence Score: The paper strongly advocates for using the log probability of generated tokens as a reliable indicator of an LLM's confidence and the potential for unanswerable queries. This is a key insight that drives the ProbGate mechanism.
  • Fine-tuning Enhances Distinguishability: Fine-tuning an LLM (like GPT-3.5-turbo) specifically on answerable Text2SQL pairs significantly improves its ability to differentiate between answerable and unanswerable queries, creating a more pronounced separation in log probability distributions.
  • Two-Stage Filtering is Crucial: Combining probabilistic filtering (ProbGate) with actual SQL execution and error handling (GEF) provides a robust approach. ProbGate acts as a first pass to identify likely unanswerable queries based on internal model uncertainty, while GEF catches errors that only become apparent during execution.
  • Prioritizing Reliability Over Answering: In sensitive domains like healthcare, the paper emphasizes that it's better for a model to abstain from answering than to provide incorrect information. This philosophy underpins the use of metrics like the Reliability Score (RS) and the GEF stage.
  • API-Friendly Approach: ProbGate's reliance on log probabilities makes it applicable even when direct access to model weights or internal states is not possible, making it valuable for models accessed via APIs.

Key Results

The proposed ProbGate method, when integrated into a pipeline with GPT-3.5-turbo fine-tuning and Grammatical Errors Filtering, achieved the best performance on the EHRSQL 2024 test set.

  • The final pipeline achieved a Reliability Score (RS) of 81.92 (RS(0)), 78.06 (RS(5)), and 74.21 (RS(10)), demonstrating strong performance across different penalty configurations.
  • This performance was superior to baseline models, including T5-based models and GPT-3.5-turbo with only a classifier for unanswerable questions.
  • The paper highlights that the fine-tuned GPT-3.5-turbo model, while good at generating SQL, struggled to distinguish unanswerable questions on its own. ProbGate significantly improved this capability.
  • The ablation studies showed that performance improved progressively with each stage of filtering (fine-tuning -> ProbGate -> GEF).

Limitations and Future Work

  • Parameter Sensitivity: The effectiveness of ProbGate relies on carefully chosen parameters, such as the log probability threshold and the number of lowest-k tokens to consider. These parameters might need to be re-tuned for different datasets, indicating a potential lack of universality.
  • Model Dependency: The primary model used was GPT-3.5-turbo, which is not open-source, and its weights are not publicly disclosed. The authors suggest that further experiments with open-source LLMs are needed to validate the approach more broadly.
  • Out-of-Distribution (OOD) Challenges: The current methodology might face challenges when the distribution of unanswerable questions in the test set significantly differs from that in the training set. Future work could explore OOD detection techniques for more robust filtering.
  • Ethical Considerations: The use of GPT-3.5-turbo raises ethical concerns regarding potential harmful content, unintended biases, and the risk of private data exposure during fine-tuning, especially when handling sensitive medical data. The authors emphasize the need for responsible AI practices and caution when handling sensitive data.

Summary updated

BibTeX
@inproceedings{kim2024probgate,
  author = {Sangryul Kim and Donghee Han and Sehyun Kim},
  title = {{ProbGate at EHRSQL 2024: Enhancing SQL Query Generation Accuracy through Probabilistic Threshold Filtering and Error Handling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure