Abstract
Text-To-SQL (T2S) conversion based on large language models (LLMs) has found a wide range of applications by leveraging the capabilities of LLMs in interpreting natural language query intent. Existing research focuses on empirical methods like prompt engineering and task-specific instructions, which often lack theoretical performance bounds. We propose a general, performance-guaranteed T2S enhancement approach called Actor-Critic (AC). By designing an Actor to produce SQL queries and a Critic to evaluate them iteratively, we derive expected performance improvements theoretically. Extensive experiments on the Spider benchmark with eleven LLMs demonstrate that the Actor-Critic method consistently improves T2S performance, serving as a robust general enhancement approach.
Introduction
Converting natural language questions into structured SQL queries is essential for making data accessible to non-technical users. While Large Language Models (LLMs) have shown promise in this domain, current state-of-the-art methods rely heavily on "prompt engineering"—finicky, task-specific instructions that may not generalize across different models. Furthermore, most existing methods are purely empirical.
The Actor-Critic (AC-SQL) framework addresses these limitations by introducing a model-agnostic approach inspired by the human "solving-and-verifying" process. Unlike previous methods, AC-SQL provides a theoretical "safety net," guaranteeing performance improvements under specific conditions.
The Actor-Critic Framework
The AC-SQL framework consists of two primary roles that work in an iterative loop:
- The Actor: Generates a candidate SQL query based on the natural language question and the database schema.
- The Critic: A dual-stage verification system:
- Execution Critic: Checks the syntactic validity of the SQL by running it against a database API.
- LLM Critic: Performs a semantic check to ensure the SQL matches the original intent of the question.
If the Critic rejects the query, the Actor is prompted to regenerate it. This cycle continues until the query is accepted or a maximum number of iterations is reached.
Theoretical Foundation
A key contribution of this work is the mathematical proof that the AC approach improves performance as long as the Critic is better than a random baseline. Specifically, if $q$ is the probability of accepting a wrong SQL (False Negative) and $s$ is the probability of rejecting a correct SQL (False Positive), the system is guaranteed to improve the Actor's baseline accuracy $p$ if $q + s \le 1$. This allows for a predictable performance boost regardless of the underlying model's initial capability.
Experimental Evaluation
The approach was evaluated on the Spider benchmark and its variants, Spider-DK (domain knowledge) and Spider-SYN (synonyms), using eleven different LLMs ranging from small open-source models to state-of-the-art proprietary systems.
Key Results
The Actor-Critic method demonstrated consistent gains in Execution Accuracy (EX) across all tested models. Notably, smaller models saw the most dramatic improvements, suggesting that verification is an easier task for LLMs than generation.
- LLaMA3-8B: Accuracy jumped from 32.6% to 67.7%, an absolute improvement of 35.1%.
- Gemma-7B: Improved from 30.3% to 65.3%.
- GPT-4o: Even high-performing models benefited, with GPT-4o moving from 72.1% to 77.7%, representing a significant reduction in the remaining error rate.
Discussion and Conclusion
The AC-SQL framework proves that self-correction is a viable and theoretically sound method for boosting Text-to-SQL performance. Its "orthogonality" means it can be combined with other techniques like Chain-of-Thought prompting for cumulative gains.
While the iterative nature of the loop increases latency and cost, the significant accuracy gains—especially for smaller, more efficient models—make it a compelling strategy for production environments. Future work will explore providing richer feedback from the Critic to the Actor to accelerate convergence and further minimize errors.
Summary updated