Reboost Large Language Model-based Text-to-SQL, Text-to-Python, and Text-to-Function - with Real Applications in Traffic Domain

Guanghu Sui, Zhishuai Li, Ziyue Li, Sun Yang, Jingqing Ruan, Hangyu Mao, Rui ZhaoarXiv 2023 · DOI · arXiv

Abstract

Large Language Models (LLMs) have demonstrated remarkable proficiency in translating natural language to SQL on academic benchmarks like Spider. However, their performance often degrades significantly when applied to complex, high-density industrial databases. This paper presents ReBoostSQL, a multi-stage prompting framework tailored for real-world business applications, specifically within the traffic management domain. ReBoostSQL addresses challenges such as implicit user intent, dense database schemas, and domain-specific terminology through a pipeline of query rewriting, "explain-squeeze" schema linking, and iterative execution feedback. Experimental results on a novel traffic dataset (CTraffic) show that ReBoostSQL improves execution accuracy from a baseline of 21% to 65.79%, demonstrating its effectiveness in bridging the gap between academic research and industrial utility.

Introduction

The advent of Large Language Models (LLMs) like GPT-4 has revolutionized the field of natural language interfaces for databases. On standard benchmarks such as Spider, these models achieve high accuracy, suggesting that the problem of Text-to-SQL is nearing a solution. Yet, practitioners in industrial settings—particularly in specialized fields like traffic management—find that these models often struggle with the "messy" reality of production databases. Researchers from SenseTime, Peking University, and the University of Cologne have identified that the gap between benchmark performance and real-world utility stems from the inherent complexity of business data and the ambiguity of user queries.

Challenges in Real-World Text-to-SQL

Industrial databases differ from academic ones in several critical ways. First, user intent is often implicit. A traffic operator might ask, "Which intersection is having queue spillback now?" without specifying the exact timestamp or the technical definition of "spillback" used in the database. Second, schema density is a major hurdle; while academic tables are typically narrow, business tables often contain dozens of columns, leading to "context window" exhaustion or model confusion. Finally, domain-specific knowledge—such as understanding spatial relationships like "upstream"—is often missing from general-purpose LLMs.

Figure 1: Challenges in translating natural language traffic queries to SQL, highlighting spatial, temporal, and domain-specific tokens prone to error.

The ReBoostSQL Framework

To overcome these obstacles, the authors developed ReBoostSQL, a modular framework that decomposes the Text-to-SQL task into four specialized stages:

  1. Query Rewriting: This module identifies and resolves vague temporal or spatial terms. For instance, it replaces "now" with a precise ISO-formatted timestamp derived from system metadata, ensuring the LLM has the exact values needed for SQL predicates.
  2. Explain-Squeeze Schema Linking: To manage high-density tables, the framework employs a two-step process. It first prompts the LLM to explain which data dimensions are required for the query. It then "squeezes" this reasoning into a minimal set of table-column pairs, drastically reducing the noise and token count in the final generation prompt.
  3. Metadata-Rich Generation: The system provides the LLM with comprehensive context, including column comments, data types, and representative value samples. This prevents common errors, such as misinterpreting abbreviated column names or using incorrect date formats.
  4. SQL Boosting: Recognizing that syntax does not guarantee semantic correctness, ReBoostSQL implements an iterative feedback loop. If a generated query results in a database error or an empty set—often a sign of a semantic mismatch—the error is fed back to the model for correction. The system attempts up to three iterations to produce a valid, non-empty result.

Experimental Evaluation

The framework was evaluated using three datasets: the standard Spider benchmark, a new traffic-focused dataset called CTraffic (averaging 26.2 columns per table), and CompanyZ, an anonymous business management dataset.

The results on CTraffic were particularly striking. Previous state-of-the-art methods like DIN-SQL, when paired with GPT-4, achieved only 21.05% execution accuracy. ReBoostSQL increased this figure to 65.79%. Similar improvements were observed on the CompanyZ dataset, where ReBoostSQL nearly doubled the performance of basic prompting even when using smaller, 5-billion parameter models.

Discussion and Future Directions

The research also explored alternative paradigms, such as Text-to-Python and Text-to-Function. While Python (via Pandas) is powerful for complex mathematical operations, it was found to be significantly less token-efficient than SQL for standard retrieval tasks. Text-to-Function, which maps queries to pre-defined code templates, offered the highest accuracy (up to 76%) but requires significant manual effort to maintain and scale.

Conclusion

ReBoostSQL demonstrates that the key to applying LLMs to industrial databases lies in structured, multi-stage reasoning and tight integration with database metadata. By addressing the semantic and structural complexities of real-world data, this framework provides a robust path forward for deploying natural language database interfaces in critical domains like traffic management.

Summary updated

Abstract

The previous state-of-the-art (SOTA) method achieved a remarkable execution accuracy on the Spider dataset, which is one of the largest and most diverse datasets in the Text-to-SQL domain. However, during our reproduction of the business dataset, we observed a significant drop in performance. We examined the differences in dataset complexity, as well as the clarity of questions' intentions, and assessed how those differences could impact the performance of prompting methods. Subsequently, We develop a more adaptable and more general prompting method, involving mainly query rewriting and SQL boosting, which respectively transform vague information into exact and precise information and enhance the SQL itself by incorporating execution feedback and the query results from the database content. In order to prevent information gaps, we include the comments, value types, and value samples for columns as part of the database description in the prompt. Our experiments with Large Language Models (LLMs) illustrate the significant performance improvement on the business dataset and prove the substantial potential of our method. In terms of execution accuracy on the business dataset, the SOTA method scored 21.05, while our approach scored 65.79. As a result, our approach achieved a notable performance improvement even when using a less capable pre-trained language model. Last but not least, we also explore the Text-to-Python and Text-to-Function options, and we deeply analyze the pros and cons among them, offering valuable insights to the community.
BibTeX
@misc{sui2023reboost,
  author = {Guanghu Sui and Zhishuai Li and Ziyue Li and Sun Yang and Jingqing Ruan and Hangyu Mao and Rui Zhao},
  title = {{Reboost Large Language Model-based Text-to-SQL, Text-to-Python, and Text-to-Function - with Real Applications in Traffic Domain}},
  year = {2023},
  eprint = {2310.18752},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.18752},
}
Figure