Few-Shot Semantic Parsing with Language Models Trained on Code

Richard Shin, Benjamin Van Durme

Abstract

This paper investigates the efficacy of large language models (LLMs) trained on code, specifically OpenAI’s Codex, for few-shot semantic parsing. Traditional LLMs like GPT-3 often require intermediate "canonical utterances" to bridge the gap between natural language and structured meaning representations. We demonstrate that Codex’s structural prior allows for direct mapping to complex machine logic with high accuracy, significantly reducing the need for manually designed intermediate languages.

Introduction

Semantic parsing—the translation of natural language into machine-readable meaning representations (MRs)—is a cornerstone of modern task-oriented dialogue systems. Historically, LLMs trained primarily on natural language have struggled with the rigid, nested syntax of MRs, necessitating the use of English-like "canonical utterances" as a bridge. This study explores whether models pre-trained on billions of lines of code can bypass this intermediate step and map natural language directly to executable logic.

Methodology

The researchers employ a few-shot, in-context learning approach, providing the model with 20 examples of (Natural Language, Meaning Representation) pairs. To ensure syntactic validity and improve efficiency, two key techniques are used:

  1. Constrained Decoding: Employs a Synchronous Context-Free Grammar (SCFG) or trie to restrict the model's output to valid tokens.
  2. Speculative Constrained Decoding: A technique that generates multiple potential completions simultaneously to be validated against the grammar, significantly increasing throughput for API-based models.

Datasets and Task Definition

The study evaluates performance on two major benchmarks: Overnight, which uses Lisp-like representations across eight domains, and SMCalFlow, a large-scale dataset of dataflow programs for calendar and weather tasks.

Examples of natural language, canonical utterances, and meaning representations from the SMCalFlow and Overnight datasets.

Key Insights

The Structural Prior of Code

Codex outperforms GPT-3 even on bespoke meaning representations it did not encounter during training. Its exposure to code provides a "structural prior" that makes it fundamentally better at handling the nested logic and strict syntax required for semantic parsing.

Direct Mapping Viability

For GPT-3, there is a substantial accuracy penalty when moving from canonical utterances to raw code. In contrast, Codex nearly eliminates this gap. This suggests that developers can skip the labor-intensive process of designing canonical intermediate languages.

Improved Natural Language Precision

Surprisingly, Codex also beats GPT-3 at generating natural language canonical utterances. This indicates that training on code may improve a model's ability to handle precise, structured natural language.

Results

On the Overnight dataset, Davinci Codex achieved 86% accuracy, compared to 81% for Davinci (GPT-3). The impact is even more pronounced in the complex SMCalFlow dataset, where Codex reduced the performance gap between canonical and direct mapping from 9.5% to just 1%.

Accuracy results for SMCalFlow, demonstrating how Codex closes the gap between canonical (C) and meaning representation (M) prediction.

Conclusion

By leveraging models trained on code, semantic parsing can be made more accurate and easier to implement. The ability of Codex to directly map natural language to structured logic represents a significant step forward for few-shot learning in dialogue systems. Future work will investigate the use of standard programming languages like Python as target meaning representations.

Summary updated

Abstract

Large language models can perform semantic parsing with little training data, when prompted with in-context examples. It has been shown that this can be improved by formulating the problem as paraphrasing into canonical utterances, which casts the underlying meaning representation into a controlled natural language-like representation. Intuitively, such models can more easily output canonical utterances as they are closer to the natural language used for pre-training. Recently, models also pre-trained on code, like OpenAI Codex, have risen in prominence. For semantic parsing tasks where we map natural language into code, such models may prove more adept at it. In this paper, we test this hypothesis and find that Codex performs better on such tasks than equivalent GPT-3 models. We evaluate on Overnight and SMCalFlow and find that unlike GPT-3, Codex performs similarly when targeting meaning representations directly, perhaps because meaning representations are structured similar to code in these datasets.
BibTeX
@inproceedings{shin2022few,
  author = {Richard Shin and Benjamin Van Durme},
  title = {{Few-Shot Semantic Parsing with Language Models Trained on Code}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=KoCOEKZBgXW},
}
Figure