An Actor-Critic Approach to Boosting Text-to-SQL Large Language Model

Ziyang Zheng, Haipeng Jing, Canyu Rui, Askar Hamdulla, Dong WangarXiv 2024 · OpenReview

Abstract

Text-To-SQL (T2S) conversion based on large language models (LLMs) has found a wide range of applications by leveraging the capabilities of LLMs in interpreting natural language query intent. Existing research focuses on empirical methods like prompt engineering and task-specific instructions, which often lack theoretical performance bounds. We propose a general, performance-guaranteed T2S enhancement approach called Actor-Critic (AC). By designing an Actor to produce SQL queries and a Critic to evaluate them iteratively, we derive expected performance improvements theoretically. Extensive experiments on the Spider benchmark with eleven LLMs demonstrate that the Actor-Critic method consistently improves T2S performance, serving as a robust general enhancement approach.

Introduction

Converting natural language questions into structured SQL queries is essential for making data accessible to non-technical users. While Large Language Models (LLMs) have shown promise in this domain, current state-of-the-art methods rely heavily on "prompt engineering"—finicky, task-specific instructions that may not generalize across different models. Furthermore, most existing methods are purely empirical.

The Actor-Critic (AC-SQL) framework addresses these limitations by introducing a model-agnostic approach inspired by the human "solving-and-verifying" process. Unlike previous methods, AC-SQL provides a theoretical "safety net," guaranteeing performance improvements under specific conditions.

The Actor-Critic Framework

The AC-SQL framework consists of two primary roles that work in an iterative loop:

  1. The Actor: Generates a candidate SQL query based on the natural language question and the database schema.
  2. The Critic: A dual-stage verification system:
  3. Execution Critic: Checks the syntactic validity of the SQL by running it against a database API.
  4. LLM Critic: Performs a semantic check to ensure the SQL matches the original intent of the question.
The Actor-Critic (AC) framework for Text-to-SQL.

If the Critic rejects the query, the Actor is prompted to regenerate it. This cycle continues until the query is accepted or a maximum number of iterations is reached.

Theoretical Foundation

A key contribution of this work is the mathematical proof that the AC approach improves performance as long as the Critic is better than a random baseline. Specifically, if $q$ is the probability of accepting a wrong SQL (False Negative) and $s$ is the probability of rejecting a correct SQL (False Positive), the system is guaranteed to improve the Actor's baseline accuracy $p$ if $q + s \le 1$. This allows for a predictable performance boost regardless of the underlying model's initial capability.

Experimental Evaluation

The approach was evaluated on the Spider benchmark and its variants, Spider-DK (domain knowledge) and Spider-SYN (synonyms), using eleven different LLMs ranging from small open-source models to state-of-the-art proprietary systems.

Key Results

The Actor-Critic method demonstrated consistent gains in Execution Accuracy (EX) across all tested models. Notably, smaller models saw the most dramatic improvements, suggesting that verification is an easier task for LLMs than generation.

Performance improvement across various LLMs using the Actor-Critic method.
  • LLaMA3-8B: Accuracy jumped from 32.6% to 67.7%, an absolute improvement of 35.1%.
  • Gemma-7B: Improved from 30.3% to 65.3%.
  • GPT-4o: Even high-performing models benefited, with GPT-4o moving from 72.1% to 77.7%, representing a significant reduction in the remaining error rate.

Discussion and Conclusion

The AC-SQL framework proves that self-correction is a viable and theoretically sound method for boosting Text-to-SQL performance. Its "orthogonality" means it can be combined with other techniques like Chain-of-Thought prompting for cumulative gains.

While the iterative nature of the loop increases latency and cost, the significant accuracy gains—especially for smaller, more efficient models—make it a compelling strategy for production environments. Future work will explore providing richer feedback from the Critic to the Actor to accelerate convergence and further minimize errors.

Summary updated

Abstract

Text-To-SQL (T2S) conversion based on large language models (LLMs) has found a wide range of applications, by leveraging the capabilities of LLMs in interpreting the query intent expressed in natural language. Existing research focuses on suitable representations for data schema and/or questions, task-specific instructions and representative examples, and complicated inference pipelines. All these methods are empirical and task specific, without a theoretical bound on performance. In this paper, we propose a simple, general, and performance guaranteed T2S enhancement approach called Actor-Critic (AC). Specifically, we design two roles using the same LLM: an Actor to produce SQL queries and a Critic to evaluate the produced SQL. If the Critic believes the produced SQL is wrong, it notifies the Actor to reproduce the SQL and perform evaluation again. By this simple iterative process, expected performance can be derived in theory. We conducted extensive experiments on the Spider and related datasets with eleven LLMs, and demonstrated that the Actor-Critic method consistently improves the performance of T2S, thus serving as a general enhancement approach for T2S conversion.
BibTeX
@misc{zheng2024actor,
  author = {Ziyang Zheng and Haipeng Jing and Canyu Rui and Askar Hamdulla and Dong Wang},
  title = {{An Actor-Critic Approach to Boosting Text-to-SQL Large Language Model}},
  year = {2024},
  eprint = {2410.22082},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.22082},
}
Figure