Grounded Adaptation for Zero-shot Executable Semantic Parsing

Victor Zhong, Mike Lewis, Sida I. Wang, Luke Zettlemoyer

Abstract

We propose Grounded Adaptation for Zero-shot Executable Semantic Parsing (GAZP) to adapt an existing semantic parser to new environments (e.g. new database schemas). GAZP combines a forward semantic parser with a backward utterance generator to synthesize data (e.g. utterances and SQL queries) in the new environment, then selects cycle-consistent examples to adapt the parser. Unlike data-augmentation, which typically synthesizes unverified examples in the training environment, GAZP synthesizes examples in the new environment whose input-output consistency are verified. On the Spider, Sparc, and CoSQL zero-shot semantic parsing tasks, GAZP improves logical form and execution accuracy of the baseline parser. Our analyses show that GAZP outperforms data-augmentation in the training environment, performance increases with the amount of GAZP-synthesized data, and cycle-consistency is central to successful adaptation.

Introduction

Semantic parsing—the process of converting natural language questions into executable code like SQL—is a cornerstone of natural language interfaces to databases. However, modern parsers face a significant "generalization gap." A model trained on a specific set of databases (e.g., academic records) often fails when deployed in a new environment with a different schema (e.g., sales or healthcare). This is because the model has not encountered the specific table names, column headers, or relational structures of the new domain.

The ideal solution is "zero-shot" adaptation: enabling a model to work in a new environment without requiring expensive, manually labeled data. While traditional data augmentation techniques exist, they typically synthesize data within the original training environment, failing to capture the nuances of the target domain. We introduce GAZP, a framework that leverages the execution environment itself to verify and adapt to new database schemas.

Methodology: The GAZP Framework

GAZP adapts a parser to a new inference environment through a "round-trip" verification process grounded in execution. The framework consists of four primary stages:

  1. Query Sampling: The system samples logical forms (SQL queries) that are valid for the new database schema. To ensure these queries are structurally realistic, it utilizes "coarse templates" derived from the training data.
  2. Backward Utterance Generation: A generator model takes these sampled SQL queries and translates them into natural language questions.
  3. Forward Semantic Parsing: The existing parser takes these generated questions and attempts to parse them back into SQL.
  4. Cycle-Consistency Grounding: This is the core innovation. The system checks if the original sampled SQL and the newly parsed SQL are equivalent. Crucially, this is not a simple string comparison; the system verifies if both queries execute to the same result on the target database.
Overview of the GAZP framework showing the cycle-consistency loop between the training and inference environments.

If the round-trip is successful, the resulting (Question, SQL) pair is deemed high-quality and added to a synthetic dataset used to fine-tune the parser for the new environment. This approach ensures that the synthetic data is both linguistically plausible and technically correct within the new schema.

Experimental Evaluation

We evaluated GAZP across three major zero-shot Text-to-SQL benchmarks that represent different levels of complexity:

  • Spider: A large-scale, cross-domain dataset focusing on complex SQL queries.
  • Sparc: A dataset designed for multi-turn, context-dependent semantic parsing.
  • CoSQL: A conversational benchmark that includes dialogue state tracking and interactive queries.
Examples of the complex, cross-domain, and multi-turn tasks found in the Spider and CoSQL benchmarks.

Key Results

GAZP consistently outperformed baseline parsers across all datasets and metrics:

  • Accuracy Improvements: On the Spider test set, GAZP improved Exact Match (EM) accuracy from 52.1% to 53.3% and Execution Accuracy (EX) from 49.8% to 53.5%.
  • Handling Complexity: The gains were most significant on "Hard" and "Extra Hard" queries. For instance, on CoSQL, GAZP improved execution accuracy on Hard queries by over 14% compared to the baseline.
  • Multi-turn Robustness: In conversational datasets like Sparc and CoSQL, GAZP demonstrated significant improvements in later dialogue turns, where error propagation typically degrades performance.

Discussion and Insights

Our analysis reveals several critical insights into why grounded adaptation works. First, execution is the ultimate truth: by using database results to verify synthetic data, GAZP avoids the noise inherent in standard augmentation. Second, environment-specific synthesis is vital; generating data directly in the target domain is far more effective than augmenting the training domain.

Furthermore, the cycle-consistency filter is essential. We found that this process prunes approximately 60% of synthesized examples. Without this filter, low-quality synthetic data actually harms model performance. Finally, we observed that performance continues to scale with the volume of cycle-consistent data, showing diminishing returns only after roughly 40,000 synthetic examples.

Conclusion and Future Work

GAZP demonstrates that semantic parsers can successfully adapt to new environments without human intervention by leveraging the grounding provided by database execution. While the current implementation relies on a reasonably high-quality initial backward generator and existing query templates, future work could explore expanding template coverage and applying this "grounded adaptation" to other domains, such as robotics or virtual assistants, where simulators can provide the necessary execution feedback.

Summary updated

Abstract

We propose Grounded Adaptation for Zero-shot Executable Semantic Parsing (GAZP) to adapt an existing semantic parser to new environments (e.g. new database schemas). GAZP combines a forward semantic parser with a backward utterance generator to synthesize data (e.g. utterances and SQL queries) in the new environment, then selects cycle-consistent examples to adapt the parser. Unlike data-augmentation, which typically synthesizes unverified examples in the training environment, GAZP synthesizes examples in the new environment whose input-output consistency are verified. On the Spider, Sparc, and CoSQL zero-shot semantic parsing tasks, GAZP improves logical form and execution accuracy of the baseline parser. Our analyses show that GAZP outperforms data-augmentation in the training environment, performance increases with the amount of GAZP-synthesized data, and cycle-consistency is central to successful adaptation.
BibTeX
@inproceedings{zhong2020grounded,
  author = {Victor Zhong and Mike Lewis and Sida I. Wang and Luke Zettlemoyer},
  title = {{Grounded Adaptation for Zero-shot Executable Semantic Parsing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=RnpD5IExm4},
}
Figure