Abstract
High-quality SQL corpora are essential for intelligent database systems. For example, Text-to-SQL requires SQL queries and corresponding natural language questions as training samples. However, collecting such query corpora remains challenging in practice due to the high cost of manual annotation, highlighting the importance of automatic SQL generation. Despite recent advances, existing generation methods still face limitations in achieving both diversity and cost-effectiveness. Furthermore, many methods treat all tables equally, overlooking schema complexity and leading to under-utilization of structurally rich tables. To address these issues, this paper proposes a multi-agent framework for high-quality and large-scale SQL generation, dubbed SQL-Factory. It decomposes the generation process into three collaborative teams: the Generation Team explores diverse query structures using a powerful language model, the Expansion Team scales promising patterns via a lightweight language model, and the Management Team adaptively schedules the workflow and evaluates the quality of synthesized queries. This modular framework ensures a balanced trade-off between diversity, scalability, and generation cost. We apply SQL-Factory to four widely used benchmarks and generate over 300,000 SQL queries with less than $200 API cost. Our generated queries achieve higher diversity compared to other methods, and extensive experiments demonstrate that the generated queries significantly improve model performance in various downstream tasks.
Introduction
Building intelligent database systems—those capable of estimating query costs, tuning configurations, or translating natural language to SQL—requires massive amounts of training data. However, high-quality SQL corpora are notoriously difficult to acquire. Manual annotation is prohibitively expensive, and existing automated methods often fall short. Template-based methods are structurally repetitive, grammar-based random generators often produce semantically meaningless queries, and LLM-based generators are either too expensive or suffer from "mode collapse," where the model repeatedly generates the same few patterns.
Furthermore, most generators treat all database tables equally, often neglecting the "structurally rich" parts of a schema, such as complex join tables, in favor of simpler, more frequent ones. SQL-Factory addresses these gaps by orchestrating a specialized multi-agent system.
Methodology: The SQL-Factory Framework
SQL-Factory treats SQL generation as a collaborative team effort, breaking the process into three specialized units:
- The Generation Team (Exploration): This team uses a powerful LLM (e.g., GPT-4o) to discover novel query structures. A Table Selection Agent analyzes the database schema to prioritize underutilized or complex tables, ensuring the model explores the structurally rich segments of the database.
- The Expansion Team (Exploitation): To achieve volume without high API costs, this team uses a lightweight, locally deployed model (e.g., Qwen2.5-Coder-14B). It takes "seed" queries from the Generation Team and creates variants through structural rewriting, alias renaming, and logical modifications.
- The Management Team (Orchestration): Serving as the system's controller, the Management Agent dynamically decides whether to explore (Generation) or exploit (Expansion) based on real-time quality metrics.
A critical component within the Management Team is the Critical Agent, which filters out invalid or redundant queries. To ensure high diversity, it employs a Hybrid Similarity Metric. This metric moves beyond simple token-level comparison by incorporating Abstract Syntax Tree (AST) edit distance to detect when two queries are structurally identical despite different surface forms, and embedding-level analysis for semantic similarity.
Experimental Evaluation and Results
The framework was applied to four major database benchmarks—TPC-DS, IMDB, Spider, and BIRD—to generate over 300,000 queries.
Key Insights
- The "Explore-Exploit" Balance: By using expensive models only for structural "skeletons" and cheaper models for "flesh" (variants), SQL-Factory achieves massive scale at a fraction of the cost. The 300,000 queries cost less than $200 in API fees; using GPT-4o alone would have exceeded $1,000.
- Schema-Aware Planning: By calculating a "Complexity Score" for tables based on attributes and referential degrees, the framework ensures the workload reflects the database's structural richness.
- Diversity and Validity: SQL-Factory outperformed state-of-the-art generators like OmniSQL in diversity (measured by Vendi Score). Additionally, only 15.34% of generated queries returned empty result sets, indicating high semantic validity.
Downstream Impact
Fine-tuning smaller models on SQL-Factory's data led to significant gains in Text-to-SQL accuracy. For example, a Llama-1B model's accuracy on the Spider benchmark jumped from 41.3% to 70.6% after training on the synthesized corpus.
Conclusion
SQL-Factory provides a modular, cost-effective solution for large-scale SQL generation. By balancing exploration and exploitation while enforcing structural diversity through AST-level analysis, it overcomes the primary limitations of existing automated methods. Future work will explore extending the framework to handle more complex SQL dialects and improving the "self-evolution" of generation prompts.
Summary updated