Grounded Adaptation for Zero-shot Executable Semantic Parsing

Victor Zhong, Mike Lewis, Sida I. Wang, Luke Zettlemoyer

Abstract

We propose Grounded Adaptation for Zeroshot Executable Semantic Parsing (GAZP) to adapt an existing semantic parser to new environments (e.g. new database schemas). GAZP combines a forward semantic parser with a backward utterance generator to synthesize data (e.g. utterances and SQL queries) in the new environment, then selects cycle-consistent examples to adapt the parser. Unlike data-augmentation, which typically synthesizes unverified examples in the training environment, GAZP synthesizes examples in the new environment whose input-output consistency are verified through execution. On the Spider, Sparc, and CoSQL zero-shot semantic parsing tasks, GAZP improves logical form and execution accuracy of the baseline parser. Our analyses show that GAZP outperforms data-augmentation in the training environment, performance increases with the amount of GAZP-synthesized data, and cycle-consistency is central to successful adaptation.

Introduction

Semantic parsers serve as a bridge between human language and machine-executable code, translating natural language questions into structured queries like SQL. While modern models achieve high performance on familiar database schemas, they often struggle when deployed to entirely new environments—a scenario known as zero-shot adaptation. Traditional approaches to this problem involve extensive human labeling or simple data augmentation in the training environment. However, these methods fail to capture the specific vocabulary and structural nuances of a target database schema.

The GAZP Framework

To address these challenges, we introduce GAZP (Grounded Adaptation for Zero-shot Executable Semantic Parsing). GAZP is designed to adapt a parser to a new environment without requiring any human-labeled data by leveraging a closed-loop synthesis process.

Overview of the GAZP framework showing the cycle-consistent adaptation loop.

The GAZP framework operates through four primary stages:

  1. Query Sampling: The system analyzes the target database schema and samples valid SQL queries based on a predefined grammar.
  2. Backward Generation: A backward utterance generator takes these sampled SQL queries and generates corresponding natural language questions.
  3. Cycle-Consistency Filtering: This critical step verifies the quality of the generated pairs. A forward parser attempts to translate the generated questions back into SQL. If the resulting SQL query produces the same execution result from the database as the original sampled query, the pair is deemed "cycle-consistent."
  4. Adaptation: The parser is then fine-tuned using a combination of the original training data and the newly verified, environment-specific synthetic data.

Model Architecture

The underlying architecture of GAZP utilizes state-of-the-art NLP components to handle the complexities of cross-domain parsing. Both the forward parser and the backward generator utilize BERT for robust encoding of schema descriptions and utterances.

Architecture of the forward semantic parser using BERT and pointer-decoders.

As shown in the forward parser architecture, the model employs a pointer-decoder mechanism. This allows the system to effectively "point" to specific table and column names within the new environment's schema, facilitating the generation of accurate logical forms even in unfamiliar domains. The use of an environment description BiLSTM ensures that the model maintains context across various schema elements.

Experimental Evaluation

We evaluated GAZP on three prominent cross-domain text-to-SQL benchmarks: Spider (complex single-turn), Sparc (multi-turn conversational), and CoSQL (complex dialogue state tracking). These datasets are particularly challenging because the test databases are entirely disjoint from the training set.

GAZP consistently outperformed baseline parsers across all metrics:

  • Spider: Exact Match (EM) accuracy improved from 56.8% to 59.1%.
  • Sparc: Execution (EX) accuracy rose from 44.0% to 47.8%.
  • CoSQL: Execution accuracy increased from 36.6% to 38.8%.

Notably, the performance gains were most significant for "Hard" and "Extra Hard" query types, suggesting that synthetic data is particularly helpful for learning complex joins and nested structures in new domains.

Key Insights and Conclusion

Our research highlights several critical findings for zero-shot adaptation. First, environment grounding is essential; adapting to the target schema is significantly more effective than general data augmentation. Second, execution as ground truth provides a powerful, objective verification mechanism that is unique to executable semantic parsing. Finally, the cycle-consistency check is the linchpin of the system; without it, the inclusion of unverified synthetic data can actually degrade model performance by introducing noise.

GAZP demonstrates that by "practicing" in a new environment through self-generated, verified examples, semantic parsers can achieve robust zero-shot performance, paving the way for more adaptable and scalable natural language interfaces to databases.

Summary updated

Abstract

We propose Grounded Adaptation for Zeroshot Executable Semantic Parsing (GAZP) to adapt an existing semantic parser to new environments (e.g. new database schemas). GAZP combines a forward semantic parser with a backward utterance generator to synthesize data (e.g. utterances and SQL queries) in the new environment, then selects cycle-consistent examples to adapt the parser. Unlike data-augmentation, which typically synthesizes unverified examples in the training environment, GAZP synthesizes examples in the new environment whose input-output consistency are verified through execution. On the Spider, Sparc, and CoSQL zero-shot semantic parsing tasks, GAZP improves logical form and execution accuracy of the baseline parser. Our analyses show that GAZP outperforms data-augmentation in the training environment, performance increases with the amount of GAZP-synthesized data, and cycle-consistency is central to successful adaptation.
BibTeX
@inproceedings{zhong2020grounded,
  author = {Victor Zhong and Mike Lewis and Sida I. Wang and Luke Zettlemoyer},
  title = {{Grounded Adaptation for Zero-shot Executable Semantic Parsing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=RnpD5IExm4},
}
Figure