Residual Skill Optimization for Text-to-SQL Ensembles

Jiongli Zhu, Haoquan Guan, Parjanya Prajakta Prashant, Nikki Lijing Kuang, Seyedeh Baharan Khatami, Canwen Xu, Xiaodong Yu, Yingyu Lin, Zhewei Yao, Yuxiong He, Babak SalimiarXiv 2026 · arXiv

TL;DR

Text-to-SQL ensembles often underperform because their candidate generation strategies produce redundant, correlated errors. DIVSKILL-SQL solves this by iteratively optimizing distinct agent "skills" (prompt-level instructions) on the exact examples previous skills failed to solve. This training on the "residual" creates highly complementary agents that significantly boost ensemble coverage and end-to-end accuracy without any model fine-tuning.

What Problem Does This Paper Address?

Translating natural language questions into executable SQL queries is incredibly challenging when dealing with massive enterprise database schemas, complex multi-step logic, and dialect-specific syntax. To tackle this, modern systems use ensembles: they generate multiple SQL candidates and use a selector to pick the final output.

However, an ensemble's performance is strictly capped by its Pass@K—the probability that the correct query exists somewhere in the pool of $K$ generated candidates. Current systems rely on heuristic tricks like high-temperature sampling or basic prompt variations to create diversity. Unfortunately, this often causes agents to make the exact same mistakes or generate noisy, hallucinated queries, causing Pass@K performance to quickly plateau.

Methods

The authors introduce DIVSKILL-SQL, a framework that explicitly targets candidate complementarity at the prompt level.

  1. Agentic Execution: Instead of generating SQL in a single forward pass, an LLM agent uses tools to inspect schemas, sample rows, draft queries, and iteratively repair execution errors.
  2. Skills as System Prompts: A "skill" is a high-level system prompt instructing the agent how to approach the problem (e.g., "decompose the question into subqueries first" vs. "draft a quick query immediately and iterate based on error messages").
  3. Residual Optimization: Rather than trying to make every prompt globally perfect, DIVSKILL-SQL builds an ensemble sequentially. It takes a seed pool of diverse skills, evaluates the current ensemble on a training set, identifies the unsolved "residual" examples, and uses an LLM-based prompt optimizer to refine the next skill specifically to tackle those failure modes.
  4. Pairwise Candidate Selection: At test time, $K$ distinct skill-conditioned agents run in parallel. The resulting candidates are deduplicated by execution output and ranked using an exhaustive pairwise LLM judge tournament to select the final query.
Figure 1: System diagram of DIVSKILL-SQL. The left panel shows skill construction: starting from diverse strategy prompts, the system repeatedly identifies unsolved questions and refines the next skill toward those remaining cases. The right panel shows test-time execution: multiple skill-guided agents solve the same Text-to-SQL problem through different interaction patterns, producing SQL candidates that are then compared to choose the final output.

Datasets

  • Spider2-Lite: A rigorous benchmark evaluating complex SQL generation across SQLite (135 examples), Snowflake (207 examples), and BigQuery (209 examples) dialects.
  • BIRD-Critic: A SQL debugging benchmark based on PostgreSQL where the system must diagnose and fix an existing buggy query.
  • Optimization Data: Skills were optimized using only ~200 examples from proprietary Snowflake data (for Spider2-Lite) and 500 standard examples from BIRD-mini-dev (for BIRD-Critic).

Key Insights and Ideas

  • Greedy Submodular Maximization: The authors prove that optimizing a skill on the residual failures of previous skills is mathematically equivalent to a greedy approximation for maximizing Pass@K coverage.
  • Controlled Diversity over Random Noise: High-temperature sampling introduces chaotic reasoning paths and hallucinations. Prompt-driven behavioral diversity (changing how the agent thinks) yields much cleaner, more diverse candidate pools.
  • Zero-Shot Dialect & Task Transfer: Remarkably, skills optimized purely on Snowflake data transferred seamlessly to BigQuery and SQLite, and even generalized from standard text-to-SQL generation to a completely different debugging task (BIRD-Critic) without any retraining.

Key Results

  • Significant Accuracy Boosts: On Spider2-Lite, DIVSKILL-SQL improved selected accuracy over the strongest ensemble baseline (CHASE-SQL) by up to +11.1 points on Snowflake and +8.3 points on BigQuery using Claude 3.5 Sonnet / Opus 4.6.
  • Better Pass@K Efficiency: DIVSKILL-SQL achieves a much steeper Pass@K curve. To match the coverage of DIVSKILL-SQL's Pass@8, standard baselines require 3 to 8 additional candidate generations, making DIVSKILL-SQL far more cost-efficient.
Figure 2: Pass@k comparison between DIVSKILL-SQL and its variants on 100-instance subsets of a) Spider2-lite and b) Bird-Critic.
  • Fewer Hallucinations: Error diagnostics showed up to 3× fewer hallucinated schema references and unsupported function calls compared to baselines.
  • Reduced Redundancy: Trajectory analysis revealed a 19% to 28% reduction in behavioral redundancy across parallel agent runs, proving that the learned skills genuinely induce distinct reasoning paths.

Limitations and Future Work

  • The Selection Gap: While candidate generation improved significantly, a large gap remains between Pass@8 (oracle accuracy) and the final selected accuracy. The selection mechanism still occasionally misses correct queries present in the pool.
  • Quadratic Selection Cost: The pairwise LLM judge scales quadratically ($O(K^2)$) with the number of unique candidate execution outputs, which can become computationally expensive as $K$ grows.
  • Future Directions: The authors suggest integrating more advanced verifiers (like outcome or process reward models) and scaling the framework to handle more complex interactive environments, such as ambiguous user intents or multi-turn user feedback.

Summary updated

Abstract

Text-to-SQL ensembles improve over single-candidate generation by drawing multiple SQL candidates and selecting one, but their effectiveness is bounded by Pass@K, the probability that at least one of K candidates is correct. Existing methods source diversity heuristically through stochastic decoding or prompt variants, leaving candidate sets dominated by correlated failures. We present DivSkill-SQL, a residual skill optimization framework that builds complementary agentic Text-to-SQL ensembles without model fine-tuning: each new skill is optimized on examples the current skill ensemble fails on, provably targeting its marginal contribution to Pass@K. On Spider2-Lite, DivSkill-SQL improves selected accuracy by up to +11.1 points on Snowflake and +8.3 on BigQuery over the strongest ensemble baseline, with consistent gains across two base models (Opus-4.6 and GPT-5.4). Skills optimized on a single dialect transfer without retraining across dialects (Snowflake, BigQuery, SQLite) and to a different task formulation, such as BIRD-Critic (+2.6 pts). Error diagnostics show up to 3x fewer hallucinated schema references and function calls, indicating that gains come from genuinely reliable complementary skills rather than surface-form variation.
BibTeX
@misc{zhu2026residual,
  author = {Jiongli Zhu and Haoquan Guan and Parjanya Prajakta Prashant and Nikki Lijing Kuang and Seyedeh Baharan Khatami and Canwen Xu and Xiaodong Yu and Yingyu Lin and Zhewei Yao and Yuxiong He and Babak Salimi},
  title = {{Residual Skill Optimization for Text-to-SQL Ensembles}},
  year = {2026},
  eprint = {2605.21792},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.21792},
}
Figure