ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement

Zijin Hong, Hao Chen, Zheng Yuan, Qinggang Zhang, Luyao Zhuang, Qing Liao, Feiran Huang, Yangqiu Song, Xiao Huang

Abstract

Despite the remarkable performance of large language models (LLMs) in text-to-SQL (SQL generation), correctly producing SQL queries remains challenging during initial generation. The SQL refinement task is subsequently introduced to correct syntactic and semantic errors in generated SQL queries. However, existing paradigms face two major limitations: (i) self-debugging becomes increasingly ineffective as modern LLMs rarely produce explicit execution errors that can trigger debugging signals; (ii) self-correction exhibits low detection precision due to the lack of explicit error modeling grounded in the question and schema, and suffers from severe hallucination that frequently corrupts correct SQLs. In this paper, we propose ErrorLLM, a framework that explicitly models text-to-SQL Errors within a dedicated LLM for text-to-SQL refinement. Specifically, we represent the user question and database schema as structural features, employ static detection to identify execution failures and surface mismatches, and extend ErrorLLM's semantic space with dedicated error tokens that capture categorized implicit semantic error types. Through a well-designed training strategy, we explicitly model these errors with structural representations, enabling the LLM to detect complex implicit errors by predicting dedicated error tokens. Guided by the detected errors, we perform error-guided refinement on the SQL structure by prompting LLMs. Extensive experiments demonstrate that ErrorLLM achieves the most significant improvements over backbone initial generation. Further analysis reveals that detection quality directly determines refinement effectiveness, and ErrorLLM addresses both sides by high detection F1 score while maintain refinement effectiveness.

Introduction

Translating natural language into SQL (text-to-SQL) is a complex task that even state-of-the-art Large Language Models (LLMs) like GPT-4o struggle with when faced with intricate database schemas. To address initial generation failures, researchers have turned to refinement techniques. However, traditional approaches like self-debugging (reacting to execution errors) and self-correction (prompting the model to "fix" its own output) are reaching their limits.

Comparison of refinement paradigms: ErrorLLM vs. traditional self-debugging and self-correction.

Modern LLMs have become so syntactically proficient that they rarely produce explicit execution errors, leaving self-debugging with no signal to act upon. Conversely, self-correction often suffers from high hallucination rates; models frequently "hallucinate" errors in perfectly valid SQL, leading to "corruption" where working code is broken. ErrorLLM addresses these issues by treating error detection as a grounded, explicit modeling task.

Methodology

ErrorLLM introduces a multi-stage framework that treats error detection as a first-class citizen. The process is divided into three core components: structural representation, explicit error modeling, and error-guided refinement.

The ErrorLLM framework: Modeling, Detecting, Localizing, and Refining SQL errors.

Structural Representation and Error Modeling

Instead of processing flat text, ErrorLLM utilizes a Question-Schema Structure (QSS) graph and represents SQL queries as Abstract Syntax Trees (ASTs). This structural approach allows the model to better understand the logical relationships between the user's question and the database schema.

A key innovation is the extension of the LLM’s vocabulary with dedicated error tokens (e.g., [Err]1 for Attribute Mismatch, [Err]7 for Value Error). The model is trained to predict these tokens based on the structural context. Detection is hierarchical: a "static" stage catches obvious execution failures, while a "semantic" stage uses the LLM to identify subtle logic errors.

Error-Guided Refinement

Once an error is detected, a localization model (LocLLM) identifies the specific nodes in the AST that require correction. Finally, a refinement model (RefLLM) uses this diagnostic data to perform a targeted fix. Crucially, the model can also predict a "No-Error" token ([Err]∅), which prevents the system from corrupting queries that are already correct.

Experimental Results

ErrorLLM was evaluated on major benchmarks including BIRD and Spider, as well as specialized variants like Spider-Realistic and Spider-DK.

  • Significant Accuracy Gains: On the BIRD benchmark, ErrorLLM improved the execution accuracy of GPT-4o from 55.87% to 66.23%, representing a relative improvement of over 18%.
  • Superior Detection: The framework achieved a 78.12 D-F1 score in error detection, significantly outperforming standard self-correction methods which typically struggle to exceed an F1 score of 40.
  • Robustness against Corruption: By explicitly identifying correct queries, ErrorLLM avoids the performance degradation often seen in other refinement agents that attempt to fix non-existent errors.

Conclusion

ErrorLLM demonstrates that explicit error modeling is essential for effective text-to-SQL refinement. By diagnosing specific issues before attempting a fix, the framework achieves higher precision and avoids the pitfalls of hallucination-driven corruption. Future work will focus on expanding the error taxonomy to include more complex SQL structures, such as subqueries and advanced operators, which were not covered in the initial training set.

Summary updated

Abstract

Despite the remarkable performance of large language models (LLMs) in text-to-SQL (SQL generation), correctly producing SQL queries remains challenging during initial generation. The SQL refinement task is subsequently introduced to correct syntactic and semantic errors in generated SQL queries. However, existing paradigms face two major limitations: (i) self-debugging becomes increasingly ineffective as modern LLMs rarely produce explicit execution errors that can trigger debugging signals; (ii) self-correction exhibits low detection precision due to the lack of explicit error modeling grounded in the question and schema, and suffers from severe hallucination that frequently corrupts correct SQLs. In this paper, we propose ErrorLLM, a framework that explicitly models text-to-SQL Errors within a dedicated LLM for text-to-SQL refinement. Specifically, we represent the user question and database schema as structural features, employ static detection to identify execution failures and surface mismatches, and extend ErrorLLM's semantic space with dedicated error tokens that capture categorized implicit semantic error types. Through a well-designed training strategy, we explicitly model these errors with structural representations, enabling the LLM to detect complex implicit errors by predicting dedicated error tokens. Guided by the detected errors, we perform error-guided refinement on the SQL structure by prompting LLMs. Extensive experiments demonstrate that ErrorLLM achieves the most significant improvements over backbone initial generation. Further analysis reveals that detection quality directly determines refinement effectiveness, and ErrorLLM addresses both sides by high detection F1 score while maintain refinement effectiveness.
BibTeX
@misc{hong2026errorllm,
  author = {Zijin Hong and Hao Chen and Zheng Yuan and Qinggang Zhang and Luyao Zhuang and Qing Liao and Feiran Huang and Yangqiu Song and Xiao Huang},
  title = {{ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement}},
  year = {2026},
  eprint = {2603.03742},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.03742},
}
Figure