Abstract
In the text-to-SQL task, sequence-to-sequence (seq-to-seq) models often lead to sub-optimal performance due to limitations in their architecture. In this paper, we present a simple yet effective approach that adapts transformer-based seq-to-seq models to robust text-to-SQL generation. Instead of inducing constraints to the decoder or reformatting the task as slot-filling, we propose to train seq-to-seq models with Schema-aware Denoising (SeaD). SeaD consists of two denoising objectives that train the model to either recover input or predict output from two novel erosion and shuffle noises. These denoising objectives act as auxiliary tasks for better modeling structural data in S2S generation. In addition, we propose a clause-sensitive execution-guided (EG) decoding strategy to overcome the limitations of EG decoding for generative models. Experiments show that our method improves performance in both schema linking and grammar correctness, establishing a new state-of-the-art on the WikiSQL benchmark.
Introduction
Translating natural language (NL) into SQL is a complex challenge for standard sequence-to-sequence (S2S) models. While these models excel at natural language translation, SQL is a structured language characterized by flexible ordering (e.g., the order of columns in a SELECT statement), rigid grammar requirements, and the necessity for precise schema linking—mapping specific NL phrases like "how many people" to database columns like attendance.
Previous research has often addressed these challenges by building specialized decoders or using "slot-filling" sketches to constrain the output. SeaD takes a different approach, asking whether the capacity of vanilla generative S2S models has been underestimated. By introducing structural noise during training, SeaD forces a standard Transformer to master these nuances implicitly.
Methodology
SeaD is built upon a standard Transformer architecture utilizing a Hybrid Pointer Generator, which allows the model to either generate SQL keywords or copy values directly from the input question. The core innovation lies in the training objectives and the decoding strategy.
Schema-aware Denoising (SeaD)
To improve structural understanding, the model is trained with two auxiliary denoising tasks:
- Erosion: The model is presented with a corrupted database schema where columns are randomly added, dropped, or permuted. If a required column is missing, the model must predict an
<unk>(unknown) token. This forces the model to identify which schema elements are truly relevant to the query. - Shuffle: The order of entities (columns and values) in the input is randomly swapped. The model must then reconstruct the original, correct sequence, teaching it the internal relationships between data points rather than relying on positional memorization.
Clause-sensitive Execution Guided (EG) Decoding
Standard Execution Guided decoding runs generated SQL candidates against a database and discards those that return errors. SeaD enhances this by making it "clause-sensitive," specifically adjusting the beam search when dealing with WHERE clauses. This prevents the model from being misled by trivial results, such as a query that returns '0' but is semantically incorrect.
Results and Evaluation
The model was evaluated on WikiSQL, a dataset containing over 80,000 annotated question-SQL pairs across 24,241 Wikipedia tables. SeaD established a new state-of-the-art (SOTA) on the benchmark at the time of publication.
As shown in the results, SeaD achieved a Logic Form Accuracy of 87.1% on the test set, surpassing previous leaders like SDSQL and BRIDGE. When combined with clause-sensitive EG decoding, the Execution Accuracy reached 92.7%. Ablation studies further highlighted the impact of the denoising tasks; the "Erosion" task alone provided a +1.5% boost in logic form accuracy, confirming that schema-aware noise is a powerful training signal.
Conclusion and Future Work
SeaD demonstrates that well-regularized, denoised Transformers are surprisingly capable of handling structural data without the need for complex, task-specific architectures. By "breaking" the schema during training, the model is forced to perform schema linking as a survival mechanism.
While highly effective on single-table queries like those in WikiSQL, future work will focus on applying SeaD to more complex, multi-table datasets like Spider, which require joins and more sophisticated reasoning. Additionally, the authors suggest that incorporating monolingual SQL or NL corpora into the Shuffle task could further enhance the model's grasp of entity ordering without requiring additional labeled pairs.
Summary updated