SeaD: End-to-end Text-to-SQL Generation with Schema-aware Denoising

Kuan Xuan, Yongbo Wang, Yongliang Wang, Zujie Wen, Yang Dong

Abstract

In the text-to-SQL task, sequence-to-sequence (seq-to-seq) models often lead to sub-optimal performance due to limitations in their architecture. In this paper, we present a simple yet effective approach that adapts transformer-based seq-to-seq models to robust text-to-SQL generation. Instead of inducing constraints to the decoder or reformatting the task as slot-filling, we propose to train seq-to-seq models with Schema-aware Denoising (SeaD). SeaD consists of two denoising objectives that train the model to either recover input or predict output from two novel erosion and shuffle noises. These denoising objectives act as auxiliary tasks for better modeling structural data in S2S generation. In addition, we propose a clause-sensitive execution-guided (EG) decoding strategy to overcome the limitations of EG decoding for generative models. Experiments show that our method improves performance in both schema linking and grammar correctness, establishing a new state-of-the-art on the WikiSQL benchmark.

Introduction

Translating natural language (NL) into SQL is a complex challenge for standard sequence-to-sequence (S2S) models. While these models excel at natural language translation, SQL is a structured language characterized by flexible ordering (e.g., the order of columns in a SELECT statement), rigid grammar requirements, and the necessity for precise schema linking—mapping specific NL phrases like "how many people" to database columns like attendance.

Previous research has often addressed these challenges by building specialized decoders or using "slot-filling" sketches to constrain the output. SeaD takes a different approach, asking whether the capacity of vanilla generative S2S models has been underestimated. By introducing structural noise during training, SeaD forces a standard Transformer to master these nuances implicitly.

Methodology

SeaD is built upon a standard Transformer architecture utilizing a Hybrid Pointer Generator, which allows the model to either generate SQL keywords or copy values directly from the input question. The core innovation lies in the training objectives and the decoding strategy.

Schema-aware Denoising (SeaD)

To improve structural understanding, the model is trained with two auxiliary denoising tasks:

  • Erosion: The model is presented with a corrupted database schema where columns are randomly added, dropped, or permuted. If a required column is missing, the model must predict an <unk> (unknown) token. This forces the model to identify which schema elements are truly relevant to the query.
  • Shuffle: The order of entities (columns and values) in the input is randomly swapped. The model must then reconstruct the original, correct sequence, teaching it the internal relationships between data points rather than relying on positional memorization.
The SeaD framework utilizes Erosion and Shuffle denoising objectives to improve structural modeling.

Clause-sensitive Execution Guided (EG) Decoding

Standard Execution Guided decoding runs generated SQL candidates against a database and discards those that return errors. SeaD enhances this by making it "clause-sensitive," specifically adjusting the beam search when dealing with WHERE clauses. This prevents the model from being misled by trivial results, such as a query that returns '0' but is semantically incorrect.

Results and Evaluation

The model was evaluated on WikiSQL, a dataset containing over 80,000 annotated question-SQL pairs across 24,241 Wikipedia tables. SeaD established a new state-of-the-art (SOTA) on the benchmark at the time of publication.

Performance comparison of SeaD against other models on the WikiSQL benchmark.

As shown in the results, SeaD achieved a Logic Form Accuracy of 87.1% on the test set, surpassing previous leaders like SDSQL and BRIDGE. When combined with clause-sensitive EG decoding, the Execution Accuracy reached 92.7%. Ablation studies further highlighted the impact of the denoising tasks; the "Erosion" task alone provided a +1.5% boost in logic form accuracy, confirming that schema-aware noise is a powerful training signal.

Conclusion and Future Work

SeaD demonstrates that well-regularized, denoised Transformers are surprisingly capable of handling structural data without the need for complex, task-specific architectures. By "breaking" the schema during training, the model is forced to perform schema linking as a survival mechanism.

While highly effective on single-table queries like those in WikiSQL, future work will focus on applying SeaD to more complex, multi-table datasets like Spider, which require joins and more sophisticated reasoning. Additionally, the authors suggest that incorporating monolingual SQL or NL corpora into the Shuffle task could further enhance the model's grasp of entity ordering without requiring additional labeled pairs.

Summary updated

Abstract

In text-to-SQL task, seq-to-seq models often lead to sub-optimal performance due to limitations in their architecture. In this paper, we present a simple yet effective approach that adapts transformer-based seq-to-seq model to robust text-to-SQL generation. Instead of inducing constraint to decoder or reformat the task as slot-filling, we propose to train seq-to-seq model with Schema aware Denoising (SeaD), which consists of two denoising objectives that train model to either recover input or predict output from two novel erosion and shuffle noises. These denoising objectives acts as the auxiliary tasks for better modeling the structural data in S2S generation. In addition, we improve and propose a clause-sensitive execution guided (EG) decoding strategy to overcome the limitation of EG decoding for generative model. The experiments show that the proposed method improves the performance of seq-to-seq model in both schema linking and grammar correctness and establishes new state-of-the-art on WikiSQL benchmark. The results indicate that the capacity of vanilla seq-to-seq architecture for text-to-SQL may have been under-estimated.
BibTeX
@inproceedings{xuan2022sead,
  author = {Kuan Xuan and Yongbo Wang and Yongliang Wang and Zujie Wen and Yang Dong},
  title = {{SeaD: End-to-end Text-to-SQL Generation with Schema-aware Denoising}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.141},
}
Figure