Metasql: A Generate-Then-Rank Framework for Natural Language to SQL Translation

Yuankai Fan, Zhenying He, Tonghui Ren, Can Huang, Yinan Jing, Kai Zhang, X. Sean Wang

Abstract

Natural Language to SQL (NL2SQL) translation is a critical task for democratizing data access. However, current auto-regressive models often suffer from limited output diversity and a lack of global structural context. We present METASQL, a "generate-then-rank" framework designed to address these challenges. METASQL decomposes natural language queries into metadata to steer diverse SQL generation and employs a two-stage ranking pipeline to select the optimal candidate. Our experiments on the SPIDER and SCIENCEBENCHMARK datasets demonstrate significant performance gains across various base models, including a 15.3% accuracy boost for GPT-4.

Introduction

The goal of Natural Language to SQL (NL2SQL) systems is to allow users to query databases using plain language. Despite recent advancements in Large Language Models (LLMs) and specialized neural architectures, these systems often struggle with complex queries. Most existing models rely on auto-regressive decoding, which generates SQL tokens sequentially. This approach has two primary drawbacks:

  1. Lack of Output Diversity: Standard decoding methods like beam search frequently produce a set of top-K candidates that are syntactically similar but logically identical or share the same incorrect logic. If the correct query structure is not within this narrow search space, the model fails.
  2. Lack of Global Context: By making local token-level decisions, models can miss the broader structural requirements of complex SQL queries, leading to "local optima" that do not satisfy the user's intent.
Example of beam search outputs showing lack of logical diversity.

METASQL addresses these issues by shifting from a single-pass generation approach to a "generate-then-rank" framework. By explicitly using query metadata to guide the generation process, METASQL ensures a diverse set of logical candidates is explored.

Methodology

The METASQL framework consists of three main stages: Semantic Decomposition, Metadata-conditioned Generation, and a Two-stage Ranking Pipeline.

The METASQL three-stage pipeline architecture.

1. Semantic Decomposition

Instead of translating natural language directly to SQL, METASQL first extracts high-level structural information. A multi-label classifier maps the input query to specific metadata. This metadata includes operator tags (e.g., JOIN, GROUP BY, EXCEPT), a hardness value (estimated structural complexity), and a correctness indicator.

2. Metadata-conditioned Generation

The extracted metadata serves as a control signal for the underlying translation model (e.g., LGESQL or GPT-4). By conditioning generation on different valid metadata combinations, the framework forces the model to explore diverse logical paths. This prevents the "near-duplicate" problem common in standard beam search and allows the system to recover the correct query even if it wasn't the model's first choice.

3. Two-stage Ranking Pipeline

Once a diverse pool of candidate SQL queries is generated, METASQL uses a specialized ranking system to identify the best match:

  • Stage 1 (Coarse Ranking): A dual-tower BERT-based model performs a rapid initial sweep, narrowing the candidate pool down to the top 10 queries.
  • Stage 2 (Fine Ranking): This stage uses a "multi-grained" feature approach. It decomposes the SQL candidates into logical units (Projections, Predicates, Joins) and converts them into natural language descriptions. These descriptions are then compared against the original user query at a phrase level, allowing for precise semantic alignment.

Experimental Results

The framework was evaluated on two rigorous benchmarks: SPIDER, the standard for cross-domain NL2SQL, and SCIENCEBENCHMARK, which features complex scientific database schemas.

Performance Gains

METASQL consistently improved the performance of various base models:

  • LGESQL on SPIDER: Accuracy increased to 77.4% (a 2.3% improvement).
  • GPT-4 on SPIDER: Accuracy saw a dramatic jump of 15.3%, reaching 69.6%.
  • Execution Accuracy: On the SPIDER test set, LGESQL's execution match score improved by 21.5%, largely due to METASQL's superior handling of value matching during ranking.

Handling Complexity

The framework proved particularly effective for complex queries. On the SDSS database within ScienceBenchmark, METASQL helped the LGESQL model triple its accuracy from 4.0% to 12.0%. This highlights the framework's ability to navigate deep join paths and nested conditions that typically baffle standard models.

Discussion and Conclusion

The success of METASQL lies in its use of metadata as a "steering wheel." By explicitly guiding the model to explore specific logical structures (e.g., "try a version with an EXCEPT clause"), the framework circumvents common parsing errors. Furthermore, the phrase-level ranking in Stage 2 allows the system to catch subtle mismatches—like a wrong table join or a missing filter—that holistic semantic models often overlook.

While METASQL represents a significant step forward, challenges remain. The system's performance is still dependent on the initial metadata classifier; if the classifier misses a critical tag, the generation stage may fail to produce the correct query. Future work will focus on improving "join path" discovery in massive schemas and exploring the use of LLMs to generate more flexible natural language descriptions for the ranking stage.

In conclusion, METASQL demonstrates that a generate-then-rank approach, powered by structural metadata, is a robust strategy for enhancing the reliability and accuracy of NL2SQL systems.

Summary updated

BibTeX
@inproceedings{fan2024metasql,
  author = {Yuankai Fan and Zhenying He and Tonghui Ren and Can Huang and Yinan Jing and Kai Zhang and X. Sean Wang},
  title = {{Metasql: A Generate-Then-Rank Framework for Natural Language to SQL Translation}},
  booktitle = {ICDE},
  year = {2024},
  doi = {10.1109/icde60146.2024.00143},
}
Figure