DataGpt-SQL-7B: An Open-Source Language Model for Text-to-SQL

Lixia Wu, Peng Li, Junhong Lou, Lei Fu

TL;DR

DataGpt-SQL-7B is a compact, open-source language model designed to turn natural language into precise SQL queries. By combining clever data augmentation, Direct Preference Optimization (DPO), and an automated "self-refine" debugging loop, this 7-billion parameter model achieves a staggering 87.2% execution accuracy on the Spider-dev benchmark, outperforming much larger proprietary models like GPT-4.

What Problem Does This Paper Address?

Translating a human question (e.g., "Which singers are older than 25?") into a functional SQL query is a cornerstone of democratizing data access. While massive, closed-source Large Language Models (LLMs) like GPT-4 are quite good at this, they come with high costs, privacy concerns, and "black box" limitations.

Smaller, open-source models often struggle with schema linking—the ability to correctly identify which specific tables and columns in a complex database relate to the user's question. They also frequently hallucinate table names or fail to follow strict SQL syntax. The authors aim to prove that a specialized, smaller model can overcome these hurdles and match the performance of the giants.

Methods

The researchers built DataGpt-SQL-7B using a multi-stage training and refinement pipeline:

  1. Base Model & Fine-Tuning: They started with CodeQwen1.5-7B-Chat and fine-tuned it on a curated dataset of over 20,000 Text-to-SQL pairs.
  2. Robust Data Augmentation: To improve schema linking, they used two novel methods:
  3. Cross-DB Augmentation: They injected "noise" by adding random tables from other databases into the prompt, forcing the model to learn how to ignore irrelevant information.
  4. Inner-DB Augmentation: They randomly added or removed unused tables and columns from the current database to ensure the model could precisely select the correct schema from a crowded context.
  5. Direct Preference Optimization (DPO): To reduce common errors (like wrong column names or missing quotes), they generated multiple SQL responses for the same question. Responses that failed execution were labeled "rejected," while successful ones were "chosen," training the model to prefer valid, executable code.
  6. The Reflection Agent (Self-Refine): They implemented a feedback loop where an "Invalid Check" component tests the generated SQL against a database. If it fails, a "SQL-Debugger" (another LLM instance) analyzes the error and attempts to fix the code.

Datasets

  • Training: A custom-constructed dataset of over 20,000 Text-to-SQL samples and a specialized preference dataset for DPO.
  • Evaluation: The primary benchmark used was Spider-dev, a widely recognized cross-domain Text-to-SQL dataset. The paper also references the "Spider-family" (Spider-DK, Spider-Syn, etc.) and the BIRD benchmark for context on evaluation metrics.

Key Insights and Ideas

  • Learning from Noise: One of the most interesting takeaways is that intentionally cluttering the training data with irrelevant tables (Cross-DB Augmentation) actually makes the model better at finding the right ones. It trains the model's "attention" to be more selective.
  • Execution as the Ultimate Truth: By using an actual SQL executor to determine "chosen" vs. "rejected" samples for DPO, the authors move beyond simple pattern matching. The model learns what works, not just what looks like SQL.
  • The Power of the Loop: The "Self-Refine" mechanism provides a significant boost. The pure model is already strong (84.8% accuracy), but giving it a "second chance" to fix its own errors based on database feedback pushes it to the top of the leaderboard (87.2%).

Key Results

DataGpt-SQL-7B demonstrates state-of-the-art performance for its size:

  • Execution Accuracy (EX): Achieved 87.2% on Spider-dev, significantly higher than the base GPT-4 (72.9%) and specialized models like SENCE-13B (84.1%).
  • Test-Suite Accuracy (TS): Achieved 83.5%, a metric that ensures the SQL isn't just returning the right answer by coincidence but is logically sound across different data scenarios.
  • Efficiency: Even without the self-refine debugger, the "pure" model achieved 84.8% EX, proving that the fine-tuning and DPO stages are highly effective on their own.

Limitations and Future Work

While the results are impressive, the authors note that there is still room for growth. Future research will focus on:

  • Complex Structures: Improving the model’s ability to handle increasingly intricate and massive database schemas.
  • Multi-Step Reasoning: Enhancing performance on complex queries that require multiple logical steps or intermediate "thoughts" to solve.
  • Broader Benchmarks: While Spider is the gold standard, testing against even more diverse real-world datasets will be a logical next step.

Summary updated

Abstract

In addressing the pivotal role of translating natural language queries into SQL commands, we propose a suite of compact, fine-tuned models and self-refine mechanisms to democratize data access and analysis for non-expert users, mitigating risks associated with closed-source Large Language Models. Specifically, we constructed a dataset of over 20K sample for Text-to-SQL as well as the preference dateset, to improve the efficiency in the domain of SQL generation. To further ensure code validity, a code corrector was integrated into the model. Our system, DataGpt-sql, achieved 87.2\% accuracy on the spider-dev, respectively, showcasing the effectiveness of our solution in text-to-SQL conversion tasks. Our code, data, and models are available at \url{https://github.com/CainiaoTechAi/datagpt-sql-7b}
BibTeX
@misc{wu2024datagpt,
  author = {Lixia Wu and Peng Li and Junhong Lou and Lei Fu},
  title = {{DataGpt-SQL-7B: An Open-Source Language Model for Text-to-SQL}},
  year = {2024},
  eprint = {2409.15985},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.15985},
}
Figure