XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL

Yingqi Gao, Yifu Liu, Xiaoxia Li, Xiaorong Shi, Yin Zhu, Yiming Wang, Shiqi Li, Wei Li, Yuntao Hong, Zhiling Luo, Jinyang Gao, Liyu Mou, Yu LiarXiv 2024 · DOI · arXiv

Introduction

Converting natural language into structured SQL queries (NL2SQL) is a critical step in democratizing data access for non-expert users. Despite the advancements in Large Language Models (LLMs), current systems face significant challenges. Standard database representations like DDL often lack the descriptive context needed for precise column identification. Furthermore, while In-Context Learning (ICL) offers generalization, it lacks the consistency of Supervised Fine-Tuning (SFT). Finally, traditional ensemble methods often rely on simple majority voting, which fails in complex SQL tasks where the correct query may be a unique "lone wolf" among various incorrect attempts.

XiYan-SQL addresses these hurdles by introducing a novel framework that combines the precision of SFT with the creative diversity of ICL, supported by a specialized semi-structured schema representation and a sophisticated selection agent.

Methodology

The XiYan-SQL framework is structured as a multi-agent pipeline designed to maximize both reasoning depth and output consistency.

The multi-agent pipeline of XiYan-SQL, illustrating the flow from schema linking to candidate selection.

M-Schema Representation

Central to the framework is M-Schema, a semi-structured format that organizes database hierarchies using specific tokens (e.g., 【DB_ID】, # Table). Unlike traditional DDL, M-Schema explicitly includes data types, primary key markers, and concise example values. This provides the LLM with a clearer "map" of the database, reducing ambiguity during column and table selection.

Schema Linking and Generation

To manage complexity, the system employs a two-stage Schema Linking process: initial retrieval via semantic similarity followed by LLM-based selection to isolate necessary tables and columns. The Multi-Generator Ensemble then produces candidate queries using two distinct strategies:

  • SFT Generators: Models fine-tuned on basic SQL syntax and enhanced tasks like SQL-to-Question generation and SQL discrimination.
  • ICL Generators: These utilize a "skeleton similarity" strategy, masking specific entities in questions to find training examples with matching logical structures rather than just keyword overlaps.

Refinement and Selection

Generated candidates are passed to a SQL Refiner, an iterative agent that executes queries and uses error messages to self-correct. The final output is determined by a specialized Candidate Selection agent. Unlike majority voting, this agent is fine-tuned to analyze logical nuances and pick the most contextually accurate query from the ensemble.

Experimental Setup and Datasets

The framework was evaluated across four diverse benchmarks to test its cross-domain and cross-dialect capabilities:

  • Spider: A standard cross-domain SQLite dataset.
  • Bird: A large-scale benchmark focused on real-world database complexity.
  • SQL-Eval: An open-source PostgreSQL evaluation set.
  • NL2GQL: A dataset for graph databases (nGQL), testing the framework's versatility beyond relational SQL.

Key Results

XiYan-SQL achieved state-of-the-art (SOTA) performance across all major benchmarks, demonstrating significant improvements over standalone models and previous ensemble methods.

Performance comparison of XiYan-SQL against leading LLMs across four major benchmarks.
  • Bird Benchmark: Reached 75.63% execution accuracy, surpassing the previous SOTA (CHASE-SQL) by 0.84%.
  • Spider Benchmark: Achieved a record 89.65% accuracy.
  • Graph Database Versatility: On the NL2GQL benchmark, XiYan-SQL reached 41.20%, vastly outperforming GPT-4o (4.86%) and Gemini 1.5 Pro (6.60%).

Ablation studies confirmed that the M-Schema representation alone improved accuracy by approximately 2%, highlighting the importance of metadata (data types and primary keys) in guiding LLM reasoning.

Discussion and Future Work

While XiYan-SQL sets new performance records, it introduces higher computational overhead and latency due to its multi-generator and iterative refinement stages. This may limit its use in real-time applications. Future research will focus on optimizing ensemble sizes and extending the M-Schema concept to a wider variety of database dialects to improve efficiency without sacrificing accuracy.

Summary updated

BibTeX
@misc{gao2024xiyan,
  author = {Yingqi Gao and Yifu Liu and Xiaoxia Li and Xiaorong Shi and Yin Zhu and Yiming Wang and Shiqi Li and Wei Li and Yuntao Hong and Zhiling Luo and Jinyang Gao and Liyu Mou and Yu Li},
  title = {{XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL}},
  year = {2024},
  eprint = {2411.08599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.08599},
}
Figure