Abstract
This paper investigates the efficacy of large language models (LLMs) trained on code, specifically OpenAI’s Codex, for few-shot semantic parsing. Traditional LLMs like GPT-3 often require intermediate "canonical utterances" to bridge the gap between natural language and structured meaning representations. We demonstrate that Codex’s structural prior allows for direct mapping to complex machine logic with high accuracy, significantly reducing the need for manually designed intermediate languages.
Introduction
Semantic parsing—the translation of natural language into machine-readable meaning representations (MRs)—is a cornerstone of modern task-oriented dialogue systems. Historically, LLMs trained primarily on natural language have struggled with the rigid, nested syntax of MRs, necessitating the use of English-like "canonical utterances" as a bridge. This study explores whether models pre-trained on billions of lines of code can bypass this intermediate step and map natural language directly to executable logic.
Methodology
The researchers employ a few-shot, in-context learning approach, providing the model with 20 examples of (Natural Language, Meaning Representation) pairs. To ensure syntactic validity and improve efficiency, two key techniques are used:
- Constrained Decoding: Employs a Synchronous Context-Free Grammar (SCFG) or trie to restrict the model's output to valid tokens.
- Speculative Constrained Decoding: A technique that generates multiple potential completions simultaneously to be validated against the grammar, significantly increasing throughput for API-based models.
Datasets and Task Definition
The study evaluates performance on two major benchmarks: Overnight, which uses Lisp-like representations across eight domains, and SMCalFlow, a large-scale dataset of dataflow programs for calendar and weather tasks.
Key Insights
The Structural Prior of Code
Codex outperforms GPT-3 even on bespoke meaning representations it did not encounter during training. Its exposure to code provides a "structural prior" that makes it fundamentally better at handling the nested logic and strict syntax required for semantic parsing.
Direct Mapping Viability
For GPT-3, there is a substantial accuracy penalty when moving from canonical utterances to raw code. In contrast, Codex nearly eliminates this gap. This suggests that developers can skip the labor-intensive process of designing canonical intermediate languages.
Improved Natural Language Precision
Surprisingly, Codex also beats GPT-3 at generating natural language canonical utterances. This indicates that training on code may improve a model's ability to handle precise, structured natural language.
Results
On the Overnight dataset, Davinci Codex achieved 86% accuracy, compared to 81% for Davinci (GPT-3). The impact is even more pronounced in the complex SMCalFlow dataset, where Codex reduced the performance gap between canonical and direct mapping from 9.5% to just 1%.
Conclusion
By leveraging models trained on code, semantic parsing can be made more accurate and easier to implement. The ability of Codex to directly map natural language to structured logic represents a significant step forward for few-shot learning in dialogue systems. Future work will investigate the use of standard programming languages like Python as target meaning representations.
Summary updated