SeaD: End-to-end Text-to-SQL Generation with Schema-aware Denoising

Kuan Xu, Yongbo Wang, Yongliang Wang, Zihao Wang, Zujie Wen, Yang Dong

Introduction

Translating natural language (NL) questions into executable SQL queries is essential for democratizing data access. Historically, standard sequence-to-sequence (seq2seq) models have struggled with this task due to token interchangeability in SQL, a lack of inherent structural constraints, and the difficulty of mapping natural language terms to specific database schema columns (schema linking).

To circumvent these issues, many researchers shifted toward complex "sketch-based" architectures that fill pre-defined slots. SeaD (Schema-aware Denoising) challenges this trend by demonstrating that vanilla Transformer-based seq2seq models can achieve state-of-the-art results when equipped with robust training objectives designed to handle schema noise and structural logic.

Methodology

SeaD utilizes an end-to-end Transformer model with a Hybrid Pointer Generator, allowing it to either generate standard SQL keywords or copy specific column names and values directly from the input. The model's performance is primarily driven by two novel schema-aware denoising objectives:

  1. Shuffle: The order of entities in the NL question or SQL query is randomly permuted during training. This teaches the model that while certain SQL clauses are interchangeable, the underlying relationships between data entities remain constant.
  2. Erosion: The model is presented with a corrupted schema where columns are randomly dropped or added. If a required column is missing, the model must output an <unk> (unknown) token.
The Erosion denoising objective forces the model to handle missing schema information and understand dependencies.

By learning from the absence of data, the model develops a sophisticated "exception handling" mechanism, ensuring it only maps questions to columns when a valid relationship exists. Additionally, the authors introduce Clause-sensitive Execution-Guided (EG) Decoding, which dynamically adjusts the search beam size based on the complexity of the SQL clause being generated (e.g., allocating more resources to the WHERE clause than the SELECT clause).

Evaluation and Results

The model was evaluated on WikiSQL, the largest human-annotated dataset for Text-to-SQL, containing over 80,000 question-query pairs. SeaD achieved a new state-of-the-art for models not relying on massive external table content.

Performance comparison of SeaD against other state-of-the-art models on the WikiSQL benchmark.

As shown in the results table, SeaD+EG$_{CS}$ achieved a Logic Form Accuracy of 87.1% and an Execution Accuracy of 92.7% on the test set. Ablation studies confirmed the importance of the denoising tasks; removing the "Erosion" objective alone resulted in a 1.5% drop in accuracy, highlighting its role in robust schema linking.

Discussion and Future Work

SeaD proves that the perceived limitations of seq2seq models in Text-to-SQL are not inherent to the architecture but are often a byproduct of insufficient training objectives. By introducing structural and schema-aware noise, the model learns the underlying grammar of SQL without hard-coded constraints.

While highly effective on the single-table queries of WikiSQL, a primary area for future work is scaling this denoising approach to multi-table "Join" queries found in more complex benchmarks like Spider. The authors also suggest that "noise re-weighting"—dynamically adjusting the difficulty of denoising tasks during training—could further enhance model robustness.

Summary updated

Abstract

In text-to-SQL task, seq-to-seq models often lead to sub-optimal performance due to limitations in their architecture. In this paper, we present a simple yet effective approach that adapts transformer-based seq-to-seq model to robust text-to-SQL generation. Instead of inducing constraint to decoder or reformat the task as slot-filling, we propose to train seq-to-seq model with Schema aware Denoising (SeaD), which consists of two denoising objectives that train model to either recover input or predict output from two novel erosion and shuffle noises. These denoising objectives acts as the auxiliary tasks for better modeling the structural data in S2S generation. In addition, we improve and propose a clause-sensitive execution guided (EG) decoding strategy to overcome the limitation of EG decoding for generative model. The experiments show that the proposed method improves the performance of seq-to-seq model in both schema linking and grammar correctness and establishes new state-of-the-art on WikiSQL benchmark. The results indicate that the capacity of vanilla seq-to-seq architecture for text-to-SQL may have been under-estimated.
BibTeX
@inproceedings{xu2022sead,
  author = {Kuan Xu and Yongbo Wang and Yongliang Wang and Zihao Wang and Zujie Wen and Yang Dong},
  title = {{SeaD: End-to-end Text-to-SQL Generation with Schema-aware Denoising}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.141},
}
Figure