Language-to-Code Translation with a Single Labeled Example

Kaj Bostrom, Harsh Jhamtani, Hao Fang, Sam Thomson, Richard Shin, Patrick Xia, Benjamin Van Durme, Jason Eisner, Jacob AndreasEMNLP 2024 · OpenReview

Abstract

Tools for translating natural language into code promise natural, open-ended interaction with databases, web APIs, and other software systems. However, this promise is complicated by the diversity and continual development of these systems, each with its own interface and distinct set of features. Building a new language-to-code translator, even starting with a large language model (LM), typically requires annotating a large set of natural language commands with their associated programs. In this paper, we describe ICIP (In-Context Inverse Programming), a method for bootstrapping a language-to-code system using mostly (or entirely) unlabeled programs written using a potentially unfamiliar (but human-readable) library or API. ICIP uses a pre-trained LM to assign candidate natural language descriptions to these programs, then iteratively refines the descriptions to ensure global consistency. Across nine different application domains from the Overnight and Spider benchmarks and text-davinci-003 and CodeLlama-7b-Instruct models, ICIP outperforms a number of prompting baselines. Indeed, in a “nearly unsupervised” setting with only a single annotated program and 100 unlabeled examples, it achieves up to 85% of the performance of a fully supervised system.

Introduction

Translating natural language (NL) into code—often referred to as semantic parsing—is a critical component for building intuitive interfaces like Text-to-SQL systems and AI coding assistants. However, as software libraries and APIs evolve, maintaining these translators becomes a "moving target" problem. Traditional supervised learning requires large datasets of paired NL commands and code snippets, which are expensive and time-consuming to produce for every new domain. This post summarizes the paper "Language-to-Code Translation with a Single Labeled Example" by Bostrom et al., presented at EMNLP 2024.

While Large Language Models (LLMs) have shown impressive few-shot capabilities, they still require several high-quality examples to grasp the specific nuances of a new API. This research explores a more efficient path: building a high-performing translator using only a single labeled example and a collection of raw, unlabeled code snippets.

The ICIP Framework

The proposed method, In-Context Inverse Programming (ICIP), treats missing natural language commands as latent variables. It leverages the fundamental insight that most code is designed to be readable. Because LLMs have seen massive amounts of code during pre-training, they are often better at describing what an unfamiliar program does than they are at writing it from scratch.

The ICIP workflow: bootstrapping a language-to-code system from unlabeled programs using an iterative labeling and parsing process.

The ICIP process consists of two primary iterative steps:

  1. Sampling (The Inverse Step): The system takes an unlabeled program and prompts the LLM to generate a natural language description. It uses the single seed example (and any high-confidence pairs from previous rounds) as a guide for the desired style and level of detail.
  2. Filtering (The Round-Trip Step): To validate the generated labels, ICIP performs a consistency check by asking the LLM to translate the new NL command back into code. The researchers introduced two filtering strategies:
  3. Hard Round Trip (HardRT): Retains the command only if the LLM regenerates the exact original program.
  4. Max Round Trip (MaxRT): Selects the command that maximizes the probability of the original program being generated, ensuring coverage even for complex programs where exact matches are rare.

Experimental Results

ICIP was evaluated across nine domains from the Spider (complex SQL) and Overnight (e.g., Calendar, Restaurants, Blocks) benchmarks using models like text-davinci-003 and CodeLlama-7b-Instruct.

Performance Benchmarks

The results demonstrate that ICIP significantly bridges the gap between few-shot prompting and full supervision. In a "nearly unsupervised" configuration (1 labeled + 100 unlabeled examples), ICIP achieved approximately 85% of the accuracy of a system trained on 100 fully labeled examples.

Comparison of ICIP against few-shot baselines on the Overnight Blocks domain.

As shown in the table above, ICIP with a single seed example (1+100) reached 29.1% accuracy on the "Blocks" domain, substantially outperforming standard 8-shot prompting, which only reached 19.5%.

Key Insights

  • The Power of MaxRT: The MaxRT filter proved superior by providing "uniform coverage" of the program space. Unlike HardRT, which might discard difficult examples, MaxRT finds the best possible description for every program.
  • Iterative Refinement: The system exhibits a self-improving loop where labels from early rounds enhance the LLM's performance in subsequent rounds.
  • Syntax Robustness: Experiments involving mangled program syntax showed that ICIP does not merely rely on memorized patterns but actually learns the mapping in-context, proving its ability to generalize to unfamiliar syntax.

Conclusion

ICIP offers a powerful method for rapidly deploying semantic parsers in new domains with minimal human intervention. By leveraging the inherent "readability" of code to LLMs and enforcing round-trip consistency, it transforms unlabeled code into a valuable training resource. Future work may explore the application of ICIP to low-resource natural languages and the use of its synthetic data to fine-tune smaller, specialized models.

Summary updated

Abstract

Tools for translating natural language into code promise natural, open-ended interaction with databases, web APIs, and other software systems. However, this promise is complicated by the diversity and continual development of these systems, each with its own interface and distinct set of features. Building a new language-to-code translator, even starting with a large language model (LM), typically requires annotating a large set of natural language commands with their associated programs. In this paper, we describe ICIP (In-Context Inverse Programming), a method for bootstrapping a language-to-code system using mostly (or entirely) unlabeled programs written using a potentially unfamiliar (but human-readable) library or API. ICIP uses a pre-trained LM to assign candidate natural language descriptions to these programs, then iteratively refines the descriptions to ensure global consistency. Across nine different application domains from the Overnight and Spider benchmarks and text-davinci-003 and CodeLlama-7b-Instruct models, ICIP outperforms a number of prompting baselines. Indeed, in a “nearly unsupervised” setting with only a single annotated program and 100 unlabeled examples, it achieves up to 85% of the performance of a fully supervised system.
BibTeX
@inproceedings{bostrom2024language,
  author = {Kaj Bostrom and Harsh Jhamtani and Hao Fang and Sam Thomson and Richard Shin and Patrick Xia and Benjamin Van Durme and Jason Eisner and Jacob Andreas},
  title = {{Language-to-Code Translation with a Single Labeled Example}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=82vMmqWjeu},
}
Figure