Abstract
Large Language Models (LLMs) have demonstrated remarkable proficiency in translating natural language to SQL on academic benchmarks like Spider. However, their performance often degrades significantly when applied to complex, high-density industrial databases. This paper presents ReBoostSQL, a multi-stage prompting framework tailored for real-world business applications, specifically within the traffic management domain. ReBoostSQL addresses challenges such as implicit user intent, dense database schemas, and domain-specific terminology through a pipeline of query rewriting, "explain-squeeze" schema linking, and iterative execution feedback. Experimental results on a novel traffic dataset (CTraffic) show that ReBoostSQL improves execution accuracy from a baseline of 21% to 65.79%, demonstrating its effectiveness in bridging the gap between academic research and industrial utility.
Introduction
The advent of Large Language Models (LLMs) like GPT-4 has revolutionized the field of natural language interfaces for databases. On standard benchmarks such as Spider, these models achieve high accuracy, suggesting that the problem of Text-to-SQL is nearing a solution. Yet, practitioners in industrial settings—particularly in specialized fields like traffic management—find that these models often struggle with the "messy" reality of production databases. Researchers from SenseTime, Peking University, and the University of Cologne have identified that the gap between benchmark performance and real-world utility stems from the inherent complexity of business data and the ambiguity of user queries.
Challenges in Real-World Text-to-SQL
Industrial databases differ from academic ones in several critical ways. First, user intent is often implicit. A traffic operator might ask, "Which intersection is having queue spillback now?" without specifying the exact timestamp or the technical definition of "spillback" used in the database. Second, schema density is a major hurdle; while academic tables are typically narrow, business tables often contain dozens of columns, leading to "context window" exhaustion or model confusion. Finally, domain-specific knowledge—such as understanding spatial relationships like "upstream"—is often missing from general-purpose LLMs.
The ReBoostSQL Framework
To overcome these obstacles, the authors developed ReBoostSQL, a modular framework that decomposes the Text-to-SQL task into four specialized stages:
- Query Rewriting: This module identifies and resolves vague temporal or spatial terms. For instance, it replaces "now" with a precise ISO-formatted timestamp derived from system metadata, ensuring the LLM has the exact values needed for SQL predicates.
- Explain-Squeeze Schema Linking: To manage high-density tables, the framework employs a two-step process. It first prompts the LLM to explain which data dimensions are required for the query. It then "squeezes" this reasoning into a minimal set of table-column pairs, drastically reducing the noise and token count in the final generation prompt.
- Metadata-Rich Generation: The system provides the LLM with comprehensive context, including column comments, data types, and representative value samples. This prevents common errors, such as misinterpreting abbreviated column names or using incorrect date formats.
- SQL Boosting: Recognizing that syntax does not guarantee semantic correctness, ReBoostSQL implements an iterative feedback loop. If a generated query results in a database error or an empty set—often a sign of a semantic mismatch—the error is fed back to the model for correction. The system attempts up to three iterations to produce a valid, non-empty result.
Experimental Evaluation
The framework was evaluated using three datasets: the standard Spider benchmark, a new traffic-focused dataset called CTraffic (averaging 26.2 columns per table), and CompanyZ, an anonymous business management dataset.
The results on CTraffic were particularly striking. Previous state-of-the-art methods like DIN-SQL, when paired with GPT-4, achieved only 21.05% execution accuracy. ReBoostSQL increased this figure to 65.79%. Similar improvements were observed on the CompanyZ dataset, where ReBoostSQL nearly doubled the performance of basic prompting even when using smaller, 5-billion parameter models.
Discussion and Future Directions
The research also explored alternative paradigms, such as Text-to-Python and Text-to-Function. While Python (via Pandas) is powerful for complex mathematical operations, it was found to be significantly less token-efficient than SQL for standard retrieval tasks. Text-to-Function, which maps queries to pre-defined code templates, offered the highest accuracy (up to 76%) but requires significant manual effort to maintain and scale.
Conclusion
ReBoostSQL demonstrates that the key to applying LLMs to industrial databases lies in structured, multi-stage reasoning and tight integration with database metadata. By addressing the semantic and structural complexities of real-world data, this framework provides a robust path forward for deploying natural language database interfaces in critical domains like traffic management.
Summary updated