Abstract
Relational databases play an important role in business, science, and more. However, many users cannot fully unleash the analytical power of relational databases, because they are not familiar with database languages such as SQL. Many techniques have been proposed to automatically generate SQL from natural language, but they suffer from two issues: (1) they still make many mistakes, particularly for complex queries, and (2) they do not provide a flexible way for non-expert users to validate and refine incorrect queries. To address these issues, we introduce a new interaction mechanism that allows users to directly edit a step-by-step explanation of a query to fix errors. Our experiments on multiple datasets, as well as a user study with 24 participants, demonstrate that our approach can achieve better performance than multiple SOTA approaches.
Introduction
While Large Language Models (LLMs) have made massive strides in "Text-to-SQL"—the task of turning a natural language question into a database query—they still struggle with complex, nested, or ambiguous requests. For a non-technical user, an incorrect SQL query is a "black box": they can’t verify its correctness, let alone fix the code.
Current "human-in-the-loop" solutions typically offer either constrained feedback (e.g., multiple-choice questions) which is too rigid for complex logic, or open-ended natural language feedback, which is often as difficult for the AI to parse as the original request. To bridge this gap, we propose STEPS (Step-by-Step Explanations), an interactive framework that allows users to fix AI-generated SQL errors by directly editing natural language explanations.
Methodology
The STEPS framework decomposes the interaction into three core stages: Ask, Understand, and Repair. This modular approach allows the system to isolate errors and provide transparent feedback to the user.
1. Rule-based Explanation Generator
Instead of presenting raw SQL, STEPS breaks the query into individual clauses. Crucially, it reorders these steps to match the SQL execution order (starting with FROM, then WHERE, etc.) rather than the syntactic order. This makes the data flow more intuitive for human verification.
2. Hybrid Correction Model
When a user edits a step, the system employs a dual-path approach to update the SQL:
- Direct Transformation: A rule-based algorithm handles "atomic" edits (e.g., swapping column names or changing operators). This method is approximately 22,000x faster than neural inference.
- Text-to-Clause Model: For complex semantic edits, a neural model (based on the SmBoP architecture) translates the modified natural language step back into a specific SQL clause.
3. SQL Rewriting and Composition
The system stitches corrected clauses back together using specific rewriting rules to ensure syntactic validity, such as automatically adding necessary JOIN statements if a new table is introduced.
Experimental Evaluation
The researchers evaluated STEPS across several major benchmarks, including Spider (complex queries), Spider-DK/Syn (robustness), and WikiSQL (simple queries). To support the "Text-to-Clause" model, a custom training set of 83,000 SQL clause-explanation pairs was generated.
Key Results
- Simulated Performance: On the Spider benchmark, STEPS reached 97.9% accuracy, outperforming existing interactive state-of-the-art (SOTA) systems by over 31%.
- Human-in-the-Loop Study: In a study with 24 participants, users using STEPS completed tasks 2x to 4x more effectively than with previous tools within the same time limit.
- Expertise Parity: Notably, SQL novices performed nearly as well as SQL experts when using STEPS, demonstrating that the natural language interface effectively levels the playing field.
Discussion and Conclusion
The success of STEPS highlights several key insights. First, error isolation is critical; fixing a single logical step is significantly easier for both the user and the AI than regenerating a complex query from scratch. Second, presenting explanations in execution order significantly lowers the mental load for non-programmers.
While highly effective, the current system is focused on single-turn interactions and supports standard SQL keywords found in the Spider benchmark. Future work will explore incorporating dialogue history and supporting data modification operations like INSERT or UPDATE. By turning the "black box" of SQL generation into an editable, step-by-step process, STEPS empowers non-expert users to fully leverage the power of relational databases.
Summary updated