Few-Shot Semantic Parsing with Language Models Trained On Code

Richard Shin, Benjamin Van Durme

Abstract

Large language models (LLMs) can perform semantic parsing with minimal training data when prompted with in-context examples. Previous research suggested that performance improves by paraphrasing natural language into "canonical utterances"—a controlled, English-like representation that bridges the gap between human speech and machine code. This paper investigates whether models pre-trained on code, such as OpenAI Codex, can bypass this intermediate step. We evaluate Codex against GPT-3 on the Overnight and SMCalFlow benchmarks. Our findings reveal that Codex significantly outperforms text-only models, particularly when mapping natural language directly to meaning representations. Unlike GPT-3, Codex maintains high accuracy without canonical utterances, likely because the target meaning representations share structural similarities with the code used during its pre-training.

The Semantic Parsing Challenge

Semantic parsing is the task of translating natural language (e.g., "Schedule a meeting for tomorrow") into a machine-readable meaning representation (MR), such as a Lisp-like executable program. Traditionally, achieving high accuracy in this domain required massive labeled datasets.

Recent "few-shot" approaches using LLMs like GPT-3 showed promise but encountered a significant hurdle: these models often struggle to output structured code directly from natural language. To mitigate this, researchers developed "canonical utterances"—highly controlled, English-like versions of the target code. While effective for text-centric models, designing these intermediate languages is labor-intensive and domain-specific. This study explores whether code-trained models can eliminate this overhead by treating meaning representations as a form of programming language.

Examples of natural language, canonical utterances, and meaning representations across the SMCalFlow and Overnight datasets.

Methodology

The researchers compared GPT-3 (text-trained) against Codex (code-trained) using a few-shot prompting framework. The experimental setup included:

  1. In-Context Learning: The models were provided with approximately 20 examples of (Natural Language, Meaning Representation) pairs within the prompt to guide generation.
  2. Constrained Decoding: To ensure the models generated syntactically valid code, the authors employed a Context-Free Grammar (CFG) or a trie to prune invalid tokens during the decoding process.
  3. Speculative Constrained Decoding: Addressing the latency and cost of LLM API calls, the team adapted the "Synchromesh" algorithm. By sampling multiple tokens at once and truncating at the first invalid token, they achieved beam-search-like efficiency without the overhead of token-by-token network requests.

Key Insights

Codex "Speaks" Structure

The most striking finding is that Codex eliminates the performance gap between predicting canonical utterances and direct meaning representations. While GPT-3 sees a sharp decline in accuracy when asked to write raw MRs, Codex performs nearly identically on both tasks. It effectively treats the bespoke Lisp-like languages of semantic parsing benchmarks as just another programming language.

The "English" Paradox

Interestingly, Codex also proved superior at generating natural language canonical utterances compared to GPT-3. This suggests that pre-training on code enhances a model's ability to handle precise, structured logic, regardless of whether the output is code or controlled natural language.

The Importance of Constraints

Despite Codex's inherent understanding of code structure, the study found that using a CFG to prune invalid tokens and employing non-greedy search methods (like beam search) still provided a significant accuracy boost. Furthermore, the "Speculative Constrained Decoding" method allowed for high accuracy with substantially higher throughput, making it a viable strategy for real-world applications using hosted APIs.

Results and Performance

Codex demonstrated clear superiority across both benchmarks:

  • Direct Mapping: On the Overnight dataset, the accuracy gap between predicting canonical utterances versus meaning representations was 13% for GPT-3 but effectively 0% for Codex.
  • Benchmark Accuracy: On SMCalFlow, Davinci Codex achieved an accuracy of 0.355, outperforming the standard Davinci (GPT-3) model’s 0.340.
  • Model Scaling: The "Cushman" version of Codex (a smaller, faster model) outperformed the largest version of GPT-3 (Davinci) on the Overnight task (0.87 vs 0.81 accuracy), indicating that code-specific pre-training is more impactful than raw parameter count for these tasks.
Comparison of accuracy when targeting canonical utterances (C) versus meaning representations (M) for GPT-3 (Davinci) and Codex across two datasets.

Conclusion and Future Work

The study concludes that code-trained models like Codex are inherently better suited for semantic parsing than text-only models. By bridging the gap between natural language and structured representations, Codex reduces the need for complex intermediate languages. Future research may explore how these models perform when the target representations are even more closely aligned with mainstream programming languages like Python, or how advanced retrieval-augmented prompt selection could further enhance few-shot performance.

Summary updated

Abstract

Large language models can perform semantic parsing with little training data, when prompted with in-context examples. It has been shown that this can be improved by formulating the problem as paraphrasing into canonical utterances, which casts the underlying meaning representation into a controlled natural language-like representation. Intuitively, such models can more easily output canonical utterances as they are closer to the natural language used for pre-training. Recently, models also pre-trained on code, like OpenAI Codex, have risen in prominence. For semantic parsing tasks where we map natural language into code, such models may prove more adept at it. In this paper, we test this hypothesis and find that Codex performs better on such tasks than equivalent GPT-3 models. We evaluate on Overnight and SMCalFlow and find that unlike GPT-3, Codex performs similarly when targeting meaning representations directly, perhaps because meaning representations are structured similar to code in these datasets.
BibTeX
@inproceedings{shin2022few,
  author = {Richard Shin and Benjamin Van Durme},
  title = {{Few-Shot Semantic Parsing with Language Models Trained On Code}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=KoCOEKZBgXW},
}
Figure