Semi-Automatic Construction of Text-to-SQL Data for Domain Transfer

Tianyi Li, Sujian Li, Mark SteedmanACL 2021

TL;DR

This paper introduces a novel semi-automatic pipeline for generating Text-to-SQL datasets, crucial for adapting language models to new database domains. By focusing on context-dependent query construction and a refined paraphrasing process, they create data that significantly improves the performance of state-of-the-art semantic parsers on unseen domains.

What Problem Does This Paper Address?

The task of mapping natural language questions to SQL queries (Text-to-SQL) is vital for interacting with databases. While large-scale datasets like SPIDER have advanced the field, existing state-of-the-art semantic parsers struggle to generalize to new, unseen domains, experiencing a sharp drop in performance. This is largely due to the scarcity and high cost of creating high-quality, domain-specific Text-to-SQL data. Previous semi-automatic methods have faced challenges in handling the complexity and variety of realistic SQL queries, and have sometimes produced data that mismatches real-world query distributions and language styles. This paper aims to bridge this gap by providing a robust, semi-automatic method for constructing Text-to-SQL data that is both broad in semantic coverage and effective for domain transfer.

Methods

The proposed pipeline consists of three main stages:

  1. Automatic SQL Query Construction: This is the core novelty. Instead of blind enumeration, they employ a context-dependent probabilistic approach.
  2. Topic Selection: They first identify a "topic" – a relevant subset of tables in the database – using a graph-based approach inspired by PageRank and Prim's MST algorithm. This helps focus the query generation on meaningful table relationships.
  3. Context-Dependent Sampling: Queries are built by sampling from a concise SQL grammar. At each step, the choice of production rules is pruned based on the "decision history" (ancestor nodes, siblings, etc.), ensuring more realistic and less contradictory query structures. They also incorporate heuristic rules to favor "good" query patterns and avoid "bad" ones.
  4. SQL-to-Pseudo-NL Conversion: Similar to prior work, constructed SQL queries are converted into pseudo-natural language (Pseudo-NL) questions using templates. They refine this by identifying a "dominant concept" for each topic to simplify the Pseudo-NL, and by balancing sentence length and complexity for better crowd-worker comprehension.
  5. Pseudo-NL to NL Paraphrasing: Crowd workers on Amazon Mechanical Turk are employed to paraphrase the Pseudo-NL questions into fluent natural language. To improve quality and mitigate "exposure effects" (where workers mimic the rigidity of Pseudo-NL), they provide personalized examples tailored to each generated SQL query, drawing from expert-annotated data.

Datasets

  • SPIDER: The experiments are based on the SPIDER dataset.
  • New Splits: They define several new data splits for their experiments:
  • seen-domain-train-set: The original SPIDER training set.
  • novel-domain-test-set: A subset of SPIDER dev set domains for testing on unseen domains.
  • seen-domain-dev-set: A subset of SPIDER dev set domains for seen domains.
  • novel-domain-ours-set: A newly constructed dataset using their full pipeline on novel domains.
  • novel-domain-oracle-set: Entries derived from gold SPIDER SQL queries in novel domains, annotated with their SQL-to-NL method.
  • seen-domain-small-set: A smaller, randomly sampled subset of the seen-domain-train-set for comparison.

Key Insights and Ideas

  • Topic-driven Query Generation: Focusing on a "topic" (a set of related tables) before generating SQL queries is more effective than random generation, leading to more semantically coherent and realistic queries.
  • Context Matters in SQL Generation: Simply enumerating grammar rules is insufficient. Conditioning the sampling process on the history of choices (decision history) significantly improves the quality and realism of generated SQL queries.
  • Balancing Pseudo-NL Complexity: Overly granular Pseudo-NL questions can confuse annotators. A balance between sentence count and individual sentence complexity is crucial for effective crowd-paraphrasing.
  • Personalized Examples for Paraphrasing: Providing crowd workers with personalized, expert-annotated examples relevant to the specific SQL query being paraphrased is a powerful technique to improve the naturalness and accuracy of the generated NL questions.
  • BERT's Role in Domain Transfer: Models with BERT encoders show a greater ability to benefit from their semi-automatically generated data, likely because BERT's pre-training on massive text corpora makes it more adept at understanding the language style of crowd-paraphrased questions.

Key Results

  • Domain Transfer Improvement: Their semi-automatically constructed data (novel-domain-ours-set) improved the accuracy of SOTA semantic parsers (lang2logic and IRNet-BERT) on unseen domains by over 1% when used for fine-tuning.
  • Human Evaluation: Participants found their generated SQL queries to be comparable to expert-annotated ones, with a higher proportion rated as succinct, sensible, and relevant, though also slightly more complex.
  • SQL-to-NL Validity: The crowd-paraphrased questions showed good alignment with expert-annotated questions, with an average similarity score of 3.78 out of 5.
  • Comparison to Baselines: When trained from random initialization, their data (random-init-ours) performed comparably to or better than expert-annotated small datasets (random-init-train(small)), demonstrating its effectiveness even without extensive pre-training.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that their generated queries are sometimes considered more complex by human evaluators compared to expert ones.
  • They note that models without BERT encoders showed less significant gains, suggesting a dependency on the model architecture's ability to handle varied language styles.
  • The cost of human evaluation and annotation, while reduced by their semi-automatic approach, is still a factor.
  • Future Work:
    • Explore more sophisticated probabilistic models for SQL query construction.
  • Integrate their query construction method with recent advancements in neural SQL-to-NL translation for greater scalability.
  • Investigate methods to further reduce the complexity gap between their generated queries and expert ones.
  • Scale up their data generation process.

Summary updated

BibTeX
@inproceedings{li2021semi,
  author = {Tianyi Li and Sujian Li and Mark Steedman},
  title = {{Semi-Automatic Construction of Text-to-SQL Data for Domain Transfer}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure