SQL-Factory: A Multi-Agent Framework for High-Quality and Large-Scale SQL Generation

Jiahui Li, Tongwang Wu, Yuren Mao, Yunjun Gao, Yajie Feng, Huaizhong LiuarXiv 2025 · arXiv · OpenReview

Abstract

High-quality SQL corpora are essential for intelligent database systems. For example, Text-to-SQL requires SQL queries and corresponding natural language questions as training samples. However, collecting such query corpora remains challenging in practice due to the high cost of manual annotation, highlighting the importance of automatic SQL generation. Despite recent advances, existing generation methods still face limitations in achieving both diversity and cost-effectiveness. Furthermore, many methods treat all tables equally, overlooking schema complexity and leading to under-utilization of structurally rich tables. To address these issues, this paper proposes a multi-agent framework for high-quality and large-scale SQL generation, dubbed SQL-Factory. It decomposes the generation process into three collaborative teams: the Generation Team explores diverse query structures using a powerful language model, the Expansion Team scales promising patterns via a lightweight language model, and the Management Team adaptively schedules the workflow and evaluates the quality of synthesized queries. This modular framework ensures a balanced trade-off between diversity, scalability, and generation cost. We apply SQL-Factory to four widely used benchmarks and generate over 300,000 SQL queries with less than $200 API cost. Our generated queries achieve higher diversity compared to other methods, and extensive experiments demonstrate that the generated queries significantly improve model performance in various downstream tasks.

Introduction

Building intelligent database systems—those capable of estimating query costs, tuning configurations, or translating natural language to SQL—requires massive amounts of training data. However, high-quality SQL corpora are notoriously difficult to acquire. Manual annotation is prohibitively expensive, and existing automated methods often fall short. Template-based methods are structurally repetitive, grammar-based random generators often produce semantically meaningless queries, and LLM-based generators are either too expensive or suffer from "mode collapse," where the model repeatedly generates the same few patterns.

Furthermore, most generators treat all database tables equally, often neglecting the "structurally rich" parts of a schema, such as complex join tables, in favor of simpler, more frequent ones. SQL-Factory addresses these gaps by orchestrating a specialized multi-agent system.

Methodology: The SQL-Factory Framework

SQL-Factory treats SQL generation as a collaborative team effort, breaking the process into three specialized units:

  1. The Generation Team (Exploration): This team uses a powerful LLM (e.g., GPT-4o) to discover novel query structures. A Table Selection Agent analyzes the database schema to prioritize underutilized or complex tables, ensuring the model explores the structurally rich segments of the database.
  2. The Expansion Team (Exploitation): To achieve volume without high API costs, this team uses a lightweight, locally deployed model (e.g., Qwen2.5-Coder-14B). It takes "seed" queries from the Generation Team and creates variants through structural rewriting, alias renaming, and logical modifications.
  3. The Management Team (Orchestration): Serving as the system's controller, the Management Agent dynamically decides whether to explore (Generation) or exploit (Expansion) based on real-time quality metrics.
The SQL-Factory multi-agent architecture, illustrating the collaborative workflow between the Management, Generation, and Expansion teams.

A critical component within the Management Team is the Critical Agent, which filters out invalid or redundant queries. To ensure high diversity, it employs a Hybrid Similarity Metric. This metric moves beyond simple token-level comparison by incorporating Abstract Syntax Tree (AST) edit distance to detect when two queries are structurally identical despite different surface forms, and embedding-level analysis for semantic similarity.

Hybrid similarity metrics used by the Critical Agent to ensure query diversity, spanning token-level, AST-level, and embedding-level analysis.

Experimental Evaluation and Results

The framework was applied to four major database benchmarks—TPC-DS, IMDB, Spider, and BIRD—to generate over 300,000 queries.

Key Insights

  • The "Explore-Exploit" Balance: By using expensive models only for structural "skeletons" and cheaper models for "flesh" (variants), SQL-Factory achieves massive scale at a fraction of the cost. The 300,000 queries cost less than $200 in API fees; using GPT-4o alone would have exceeded $1,000.
  • Schema-Aware Planning: By calculating a "Complexity Score" for tables based on attributes and referential degrees, the framework ensures the workload reflects the database's structural richness.
  • Diversity and Validity: SQL-Factory outperformed state-of-the-art generators like OmniSQL in diversity (measured by Vendi Score). Additionally, only 15.34% of generated queries returned empty result sets, indicating high semantic validity.

Downstream Impact

Fine-tuning smaller models on SQL-Factory's data led to significant gains in Text-to-SQL accuracy. For example, a Llama-1B model's accuracy on the Spider benchmark jumped from 41.3% to 70.6% after training on the synthesized corpus.

Conclusion

SQL-Factory provides a modular, cost-effective solution for large-scale SQL generation. By balancing exploration and exploitation while enforcing structural diversity through AST-level analysis, it overcomes the primary limitations of existing automated methods. Future work will explore extending the framework to handle more complex SQL dialects and improving the "self-evolution" of generation prompts.

Summary updated

Abstract

High quality SQL corpus is essential for intelligent database. For example, Text-to-SQL requires SQL queries and correspond natural language questions as training samples. However, collecting such query corpus remains challenging in practice due to the high cost of manual annotation, which highlights the importance of automatic SQL generation. Despite recent advances, existing generation methods still face limitations in achieving both diversity and cost-effectiveness. Besides, many methods also treat all tables equally, which overlooks schema complexity and leads to under-utilization of structurally rich tables. To address these issues, this paper proposes a multi-agent framework for high-quality and large-scale SQL generation, dubbed SQL-Factory. It decomposes the generation process into three collaborative teams: the Generation Team explores diverse query structures using a powerful language model, the Expansion Team scales promising patterns via a lightweight language model, and the Management Team adaptively schedules the workflow and evaluates the quality of synthesized queries. This modular framework ensures a balanced trade-off between diversity, scalability, and generation cost. We apply SQL-Factory to four widely used benchmarks and generate over 300,000 SQL queries with less than $200 API cost. Our generated queries achieve higher diversity compared to other methods, and extensive experiments demonstrate that the generated queries significantly improve the model performance in various downstream tasks.
BibTeX
@misc{li2025sql,
  author = {Jiahui Li and Tongwang Wu and Yuren Mao and Yunjun Gao and Yajie Feng and Huaizhong Liu},
  title = {{SQL-Factory: A Multi-Agent Framework for High-Quality and Large-Scale SQL Generation}},
  year = {2025},
  eprint = {2504.14837},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.14837},
}
Figure