SAFE-SQL: Self-Augmented In-Context Learning with Fine-grained Example Selection for Text-to-SQL

Jimin Lee, Insuck Baek, B. Kim, Hyunkyung Bae, Hwanhee Lee2025 · DOI

Abstract

SAFE-SQL is a novel unsupervised framework designed to improve Text-to-SQL performance by enabling Large Language Models (LLMs) to generate their own high-quality in-context learning examples. By utilizing a fine-grained filtering mechanism based on semantic, structural, and reasoning alignment, SAFE-SQL bypasses the need for massive training sets and addresses common retrieval failures. Experimental results on industry-standard benchmarks like Spider and BIRD show that SAFE-SQL outperforms standard few-shot methods and even supervised fine-tuning on complex tasks.

Introduction

Converting natural language into executable SQL (Text-to-SQL) is essential for making data accessible to non-technical users. Currently, state-of-the-art LLMs rely on "few-shot" prompting, where they are provided with example pairs of questions and SQL queries. However, this approach faces two major hurdles: the "Cold Start" problem, where pre-existing examples for a new database are unavailable, and retrieval failure, where standard methods select examples that are superficially similar but logically different.

SAFE-SQL (Self-Augmented In-context learning with Fine-grained Example selection) addresses these challenges by allowing the model to generate its own logically relevant examples, ensuring high-quality guidance even in the absence of a curated training set.

Methodology

The SAFE-SQL framework follows a five-step unsupervised pipeline to generate and select the most effective in-context examples for a given user query.

The SAFE-SQL pipeline: from schema linking and example generation to relevance scoring, filtering, and final inference.
  1. Schema Linking: The system analyzes the user's question against the database schema to identify relevant tables, columns, and foreign keys.
  2. Example Generation: Instead of searching a static database, SAFE-SQL prompts an LLM to generate 10 synthetic "triplets" consisting of a Question, a SQL Query, and a Reasoning Path based on the identified schema elements.
  3. Relevance Scoring: Each synthetic example is scored (0-10) based on three fine-grained criteria: Semantic Similarity (intent matching), Structural Alignment (SQL logic matching), and Reasoning Path Quality (logical soundness).
  4. Threshold Filtering: Only examples scoring above a specific threshold (typically 8/10) are retained to ensure the LLM is not misled by noisy or incorrect synthetic data.
  5. Final SQL Inference: The original user question is sent to the LLM, primed with the high-quality, self-generated examples to produce the final executable SQL.

Evaluation and Results

The researchers evaluated SAFE-SQL on two major benchmarks: Spider, a large-scale cross-domain dataset, and BIRD, which emphasizes complex real-world reasoning on large databases.

Key Performance Metrics

  • State-of-the-Art Performance: On the Spider benchmark, SAFE-SQL using GPT-4o achieved an 87.9% Execution Accuracy (EX), significantly outperforming standard zero-shot (73.4%) and traditional few-shot retrieval methods like DAIL-SQL (83.6%).
  • Excelling at Complexity: The most dramatic gains occurred in the most difficult categories. On Spider's "Extra Hard" tasks, SAFE-SQL improved accuracy by nearly 12% compared to standard retrieval methods.
  • Synthetic vs. Supervised: On the BIRD dataset, SAFE-SQL achieved 63.5% execution accuracy, slightly outperforming models specifically fine-tuned on supervised data (63.4%). This suggests that context-aware synthetic prompts can be as effective as rigid fine-tuning.

Discussion and Insights

The Importance of Filtering

A critical finding of the study is the "sweet spot" for the relevance threshold. As shown in the performance analysis, a threshold of 8 provides the optimal balance. If the filter is too loose, noisy examples degrade performance; if it is too strict, the model loses the diversity of examples needed for generalization.

Impact of relevance score threshold on execution accuracy across different difficulty levels, showing the optimal performance peak at a threshold of 8.

The Role of Reasoning Paths

Including a step-by-step logical derivation (the reasoning path) in the generated examples was found to be vital for success on "Hard" and "Extra Hard" queries. These paths act as a roadmap, helping the LLM navigate complex joins and subqueries that surface-level similarity might miss.

Limitations and Future Work

While highly effective, SAFE-SQL's multi-step process (Generation, Filtering, and Inference) increases latency and computational cost compared to single-shot methods. Future research will focus on compressing this pipeline or utilizing smaller, more efficient models for the initial generation and filtering stages to make the framework more viable for real-time applications.

Conclusion

SAFE-SQL demonstrates that LLMs can effectively "help themselves" by generating and filtering their own in-context learning examples. By focusing on fine-grained logical and structural alignment rather than simple keyword similarity, the framework sets a new standard for unsupervised Text-to-SQL performance, particularly in complex, data-heavy environments.

Summary updated

BibTeX
@misc{lee2025safe,
  author = {Jimin Lee and Insuck Baek and B. Kim and Hyunkyung Bae and Hwanhee Lee},
  title = {{SAFE-SQL: Self-Augmented In-Context Learning with Fine-grained Example Selection for Text-to-SQL}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.962},
}
Figure