Abstract
The Text-to-SQL capabilities of large language models (LLMs) allow users to interact with databases using natural language. Despite recent advances, existing models continue to struggle with complex queries, particularly those requiring multi-table joins and reasoning. To address this gap, we propose SAC-SQL, a framework that utilizes synthetic training samples followed by a structure-aware curriculum learning approach. Our method begins with supervised fine-tuning (SFT) on a synthetically constructed, cross-domain SQL dataset with diverse structural patterns. We then introduce a unified structure difficulty scoring function to partition training samples into non-overlapping curriculum phases, guiding the model to learn progressively from simpler to more complex SQL structures. Extensive experiments show that SAC-SQL achieves superior results, significantly narrowing the performance gap between open-source and closed-source models on the Spider and BIRD benchmarks.
Introduction
While Large Language Models (LLMs) have made strides in translating natural language into SQL, they often struggle with "structurally complex" queries involving multi-table joins, deep nesting, and intricate reasoning (e.g., HAVING clauses combined with WINDOW functions). The primary bottlenecks identified are data scarcity—high-quality, diverse Text-to-SQL datasets are expensive to annotate—and uniform training, where models are exposed to a random mix of easy and hard samples, failing to generalize to the rare, complex patterns found in the "long tail" of SQL logic.
Methodology
The SAC-SQL (Structure-Aware Curriculum SQL) pipeline consists of two primary stages designed to mimic the human learning process.
1. Synthetic Data Generation
To overcome data scarcity, we leveraged closed-source LLMs (GPT-4 and Claude-2) to generate a massive, cross-domain dataset. This synthetic set covers over 500 distinct database schemas and explicitly prompts for underrepresented SQL constructs like JOIN, GROUP BY, and NESTED SELECT. The resulting dataset contains 12,134 examples across 674 databases, featuring a significantly higher "Nested Ratio" (18.7%) and "Avg JOIN" count (1.26) compared to standard benchmarks like Spider.
2. Structure-Aware Curriculum Learning
Instead of random sampling, the model learns in five distinct phases, moving from easy to hard. This is guided by a novel Difficulty Scoring Function $D(x)$ that considers:
- Base Structure ($S$): Weights assigned to specific SQL functions.
- Interaction ($I$): Extra difficulty points when complex components co-occur (e.g.,
JOIN+HAVING). - Syntactic Depth ($T$): The complexity of the SQL's abstract syntax tree.
- Model Uncertainty ($L$): The model's own token-level loss.
Key Results
SAC-SQL demonstrated state-of-the-art performance for open-source models across major benchmarks. On the Spider Benchmark, SAC-SQL-13B achieved 83.2% Execution Accuracy (EX), outperforming strong baselines like RESDSQL. On the more challenging BIRD Benchmark, it reached 59.7% EX on the test set, notably surpassing the zero-shot performance of GPT-4 (54.9%).
Ablation studies confirmed the necessity of the curriculum; removing this component caused a performance drop from 59.7% to 50.5% on BIRD, proving that the order of learning is as critical as the data itself.
Discussion and Limitations
While effective, the current difficulty scoring function is static and defined before training. Future work may explore dynamic functions that evolve as the model gains competence. Additionally, while synthetic data provides diversity, it may still carry biases from the generating LLMs. The authors plan to integrate execution-guided feedback—where the model learns from the results of the SQL it generates—to further refine accuracy and bridge the gap with proprietary models.
Summary updated