NL2pSQL: Generating Pseudo-SQL Queries from Under-Specified Natural Language Questions

Fuxiang Chen, Seung-won Hwang, Jaegul Choo, Jung-Woo Ha, Sunghun Kim

Introduction

Traditional Natural Language to SQL (NL2SQL) research typically operates under the assumption of a "perfect world": users provide clear, specific questions, and a complete database schema is available for execution. However, real-world developer interactions—such as those found on Stack Overflow—are often messy and "under-specified." Developers frequently seek help with SQL logic or syntax templates without providing specific table names or schemas. In these scenarios, generating a 100% executable query is impossible and often unnecessary.

This paper introduces the task of NL2pSQL, which focuses on generating human-readable Pseudo-SQL (pSQL). These queries serve as templates to guide developers, prioritizing structural correctness and logic over immediate executability.

The Challenge of Under-Specified Queries

On platforms like Stack Overflow, a question might be as vague as "How do I copy from one table to another?" A traditional NL2SQL model would struggle without a schema. NL2pSQL addresses this by producing a well-formed template like INSERT INTO table1 SELECT * FROM table2.

To support this research, the authors curated the NL2pSQL Dataset, consisting of 1,244 pairs of natural language questions and SQL answers crawled from Stack Overflow. Unlike standard datasets like WikiSQL, which focus on simple SELECT statements, this dataset covers 362 distinct SQL keywords, reflecting the diverse needs of professional developers.

Proposed Architecture: The Denoising Pipeline

The core technical contribution is a two-stage generative architecture designed to ensure that the output is not only relevant but also syntactically valid (well-formed).

  1. Initial Generation: A base sequence-to-sequence (Seq2Seq) or CopyNet model translates the natural language input into an initial SQL statement.
  2. Denoising Autoencoder (DAE): Generative models often produce "noisy" SQL characterized by repetitive tokens or syntax errors. The authors introduced a DAE trained on a massive synthetic dataset of 406,384 perturbed SQL pairs. This layer acts as a "syntax fixer," taking the potentially ill-formed output from the first stage and refining it into a well-formed statement that passes a SQL parser.
The NL2pSQL architecture uses a denoising autoencoder to transform ill-formed generated SQL into well-formed, human-readable code.

Evaluation Metrics

Because standard BLEU scores can be misleading for code—where a logically perfect query might use different variable names than the ground truth—the authors proposed two new metrics:

  • Canonical-BLEU: Replaces identifiers with indexed placeholders to focus on logic.
  • SQL-BLEU: Measures structural accuracy by calculating scores based solely on SQL keywords.

Key Results

The addition of the Denoising Autoencoder yielded significant improvements in the "well-formedness" of the generated queries. As shown in the experimental results, the percentage of parsable queries increased by as much as 43% in certain configurations (e.g., Seq2Seq Char-based models).

Performance comparison showing the significant increase in well-formedness (indicated by arrows) after applying the denoising layer across different model configurations.

Crucially, while the DAE drastically improved the syntax and readability of the SQL, the underlying BLEU scores remained stable. This indicates that the denoising process successfully repairs the "form" of the query without distorting the "meaning" captured during the initial translation phase.

Conclusion

NL2pSQL shifts the focus of natural language interfaces for databases from strict execution to developer support. By cascading a standard generator with a denoising autoencoder, the system provides a robust tool for generating high-quality SQL templates from the kind of under-specified questions common in real-world programming environments. Future work aims to expand this approach to multiple SQL dialects and incorporate broader conversational context from developer forums.

Summary updated

BibTeX
@inproceedings{chen2019nl,
  author = {Fuxiang Chen and Seung-won Hwang and Jaegul Choo and Jung-Woo Ha and Sunghun Kim},
  title = {{NL2pSQL: Generating Pseudo-SQL Queries from Under-Specified Natural Language Questions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1262},
}
Figure