TL;DR
Text-to-SQL ensembles often underperform because their candidate generation strategies produce redundant, correlated errors. DIVSKILL-SQL solves this by iteratively optimizing distinct agent "skills" (prompt-level instructions) on the exact examples previous skills failed to solve. This training on the "residual" creates highly complementary agents that significantly boost ensemble coverage and end-to-end accuracy without any model fine-tuning.
What Problem Does This Paper Address?
Translating natural language questions into executable SQL queries is incredibly challenging when dealing with massive enterprise database schemas, complex multi-step logic, and dialect-specific syntax. To tackle this, modern systems use ensembles: they generate multiple SQL candidates and use a selector to pick the final output.
However, an ensemble's performance is strictly capped by its Pass@K—the probability that the correct query exists somewhere in the pool of $K$ generated candidates. Current systems rely on heuristic tricks like high-temperature sampling or basic prompt variations to create diversity. Unfortunately, this often causes agents to make the exact same mistakes or generate noisy, hallucinated queries, causing Pass@K performance to quickly plateau.
Methods
The authors introduce DIVSKILL-SQL, a framework that explicitly targets candidate complementarity at the prompt level.
- Agentic Execution: Instead of generating SQL in a single forward pass, an LLM agent uses tools to inspect schemas, sample rows, draft queries, and iteratively repair execution errors.
- Skills as System Prompts: A "skill" is a high-level system prompt instructing the agent how to approach the problem (e.g., "decompose the question into subqueries first" vs. "draft a quick query immediately and iterate based on error messages").
- Residual Optimization: Rather than trying to make every prompt globally perfect, DIVSKILL-SQL builds an ensemble sequentially. It takes a seed pool of diverse skills, evaluates the current ensemble on a training set, identifies the unsolved "residual" examples, and uses an LLM-based prompt optimizer to refine the next skill specifically to tackle those failure modes.
- Pairwise Candidate Selection: At test time, $K$ distinct skill-conditioned agents run in parallel. The resulting candidates are deduplicated by execution output and ranked using an exhaustive pairwise LLM judge tournament to select the final query.
Datasets
- Spider2-Lite: A rigorous benchmark evaluating complex SQL generation across SQLite (135 examples), Snowflake (207 examples), and BigQuery (209 examples) dialects.
- BIRD-Critic: A SQL debugging benchmark based on PostgreSQL where the system must diagnose and fix an existing buggy query.
- Optimization Data: Skills were optimized using only ~200 examples from proprietary Snowflake data (for Spider2-Lite) and 500 standard examples from BIRD-mini-dev (for BIRD-Critic).
Key Insights and Ideas
- Greedy Submodular Maximization: The authors prove that optimizing a skill on the residual failures of previous skills is mathematically equivalent to a greedy approximation for maximizing Pass@K coverage.
- Controlled Diversity over Random Noise: High-temperature sampling introduces chaotic reasoning paths and hallucinations. Prompt-driven behavioral diversity (changing how the agent thinks) yields much cleaner, more diverse candidate pools.
- Zero-Shot Dialect & Task Transfer: Remarkably, skills optimized purely on Snowflake data transferred seamlessly to BigQuery and SQLite, and even generalized from standard text-to-SQL generation to a completely different debugging task (BIRD-Critic) without any retraining.
Key Results
- Significant Accuracy Boosts: On Spider2-Lite, DIVSKILL-SQL improved selected accuracy over the strongest ensemble baseline (CHASE-SQL) by up to +11.1 points on Snowflake and +8.3 points on BigQuery using Claude 3.5 Sonnet / Opus 4.6.
- Better Pass@K Efficiency: DIVSKILL-SQL achieves a much steeper Pass@K curve. To match the coverage of DIVSKILL-SQL's Pass@8, standard baselines require 3 to 8 additional candidate generations, making DIVSKILL-SQL far more cost-efficient.
- Fewer Hallucinations: Error diagnostics showed up to 3× fewer hallucinated schema references and unsupported function calls compared to baselines.
- Reduced Redundancy: Trajectory analysis revealed a 19% to 28% reduction in behavioral redundancy across parallel agent runs, proving that the learned skills genuinely induce distinct reasoning paths.
Limitations and Future Work
- The Selection Gap: While candidate generation improved significantly, a large gap remains between Pass@8 (oracle accuracy) and the final selected accuracy. The selection mechanism still occasionally misses correct queries present in the pool.
- Quadratic Selection Cost: The pairwise LLM judge scales quadratically ($O(K^2)$) with the number of unique candidate execution outputs, which can become computationally expensive as $K$ grows.
- Future Directions: The authors suggest integrating more advanced verifiers (like outcome or process reward models) and scaling the framework to handle more complex interactive environments, such as ambiguous user intents or multi-turn user feedback.
Summary updated