InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback

John Yang, Akshara Prabhakar, Karthik Narasimhan, Shunyu Yao

TL;DR

InterCode is a novel framework that transforms code generation from a static, one-shot task into an interactive Reinforcement Learning (RL) process. By providing Large Language Models (LLMs) with a "write-execute-test" loop via Docker-based sandboxes, the researchers demonstrate that execution feedback can dramatically boost coding performance—propelling GPT-4’s SQL success rate from a meager 9.1% to a staggering 73.7%.

What Problem Does This Paper Address?

Traditional coding benchmarks (like HumanEval or MBPP) treat code generation as a "sequence transduction" problem: a natural language instruction goes in, and a block of code comes out. If there is a typo or a logic error, the model fails.

This is not how humans code. Real-world programming is an iterative dialogue with a compiler or interpreter. We write a snippet, check the error message, inspect the environment, and refine our approach. Current benchmarks ignore this "execution feedback," leading to a disconnect between the generated code and its final environment. InterCode addresses this gap by providing a standardized, safe, and language-agnostic framework for interactive coding.

Methods

The authors frame interactive coding as a Partially Observable Markov Decision Process (POMDP).

The InterCode interactive loop and evaluation framework

In this setup:

  • Actions: The code snippets generated by the agent.
  • Observations: The execution feedback (stdout, stderr) from the environment.
  • Rewards: A score (often binary 0/1) based on whether the final state matches the "gold" requirement.

The framework is built on Docker, ensuring that code execution is safe, reproducible, and isolated from the host system. It is designed to be modular, allowing researchers to easily plug in new languages, datasets, or reward functions. The authors also evaluated several prompting strategies to see how LLMs handle this interaction:

  • Single Turn: The traditional one-shot approach.
  • Try Again: An iterative loop where the model gets $n$ chances to fix its code based on errors.
  • ReAct: A strategy where the model interleaves "Thoughts" (reasoning) with "Actions" (code).
  • Plan & Solve: The model first creates a multi-step plan and then executes it.

Datasets

InterCode doesn't just invent new data; it "interactivizes" existing popular static datasets by grounding them in real file systems and databases:

  1. InterCode-Bash: Derived from NL2Bash, featuring 200 tasks across four distinct Ubuntu file system environments.
  2. InterCode-SQL: Derived from the Spider dataset, featuring 1,034 tasks across 20 different MySQL databases.
  3. InterCode-Python: Derived from the MBPP dataset, using unit tests as the feedback mechanism.
  4. InterCode-CTF: A newly introduced "Capture the Flag" task involving multi-step security puzzles that require navigating file systems and exploiting vulnerabilities.

Key Insights and Ideas

  • Interaction as Discovery: In SQL tasks, models didn't just use feedback to fix errors; they used it for context discovery. For example, a model might run SHOW TABLES to understand the database schema before attempting the actual query.
  • Task-Specific Reasoning: Different coding tasks benefit from different "thinking" styles. The authors found that ReAct (discovery-oriented) was superior for SQL, while Plan & Solve (procedural) was more effective for Bash, where tasks are often multi-step and declarative.
  • The "Plateau" Effect: While interaction helps, models often hit a ceiling. In later turns, if a model hasn't solved the task, it tends to repeat the same mistakes or hallucinate, suggesting that current LLMs still struggle to effectively manage long-term interaction history.
  • Safety First: By using Docker, InterCode allows agents to perform "dangerous" actions like rm -rf or sudo without risking the host machine, which is essential for tasks like CTF.

Key Results

The results highlight a massive performance gap between static and interactive coding:

Success rates of various LLMs across multiple interaction turns
  • GPT-4 Dominance: On the SQL benchmark, GPT-4’s success rate jumped from 9.1% (Single Turn) to 73.7% (Try Again).
  • GPT-3.5 Improvements: Similarly, GPT-3.5 saw its SQL success rate rise from 10.5% to 47.3%.
  • Open Source Lag: While interaction helped open-source models like Vicuna and StarChat, they still lagged significantly behind proprietary models, often failing to reach even a 10% success rate on SQL.
  • Efficiency: Using ReAct allowed models to solve tasks with fewer turns and fewer errors compared to a naive "Try Again" approach, proving that how a model reasons about feedback is just as important as the feedback itself.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Language Support: Currently, the framework focuses on Bash, SQL, and Python. Expanding to languages like C++ or Java is a planned next step.
  • Dataset Scale: The CTF dataset is currently small (the paper notes a limitation of only four fully curated instances for the deep analysis) due to the high manual effort required to design these complex puzzles.
  • Human-AI Collaboration: The current framework focuses on autonomous agents. Future versions could incorporate human-in-the-loop feedback to better mirror real-world pair programming.
  • Context Windows: As interaction history grows, models hit token limits. Better memory management or retrieval-augmented generation (RAG) for interaction logs is a promising research direction.

Summary updated

Abstract

Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-to-code sequence transduction process, which has the potential for error propagation and a disconnect between the generated code and its final execution environment. To address this gap, we introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning (RL) environment, with code as actions and execution feedback as observations. Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution, and is compatible out-of-the-box with traditional seq2seq coding methods, while enabling the development of new methods for interactive code generation. We use InterCode to create three interactive code environments with Bash, SQL, and Python as action spaces, leveraging data from the static NL2Bash, Spider, and MBPP datasets. We demonstrate InterCode’s viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct and Plan & Solve. Our results showcase the benefits of interactive code generation and demonstrate that InterCode can serve as a challenging benchmark for advancing code understanding and generation capabilities. InterCode is designed to be easily extensible and can even be used to create new tasks such as Capture the Flag, a popular coding puzzle that is inherently multi-step and involves multiple programming languages.
BibTeX
@inproceedings{yang2023intercode,
  author = {John Yang and Akshara Prabhakar and Karthik Narasimhan and Shunyu Yao},
  title = {{InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=iZSFixKS2Q},
}
Figure