InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback

John Yang, Akshara Prabhakar, Karthik Narasimhan, Shunyu Yao

Abstract

Humans write code in a fundamentally interactive manner, relying on constant execution feedback to correct errors, resolve ambiguities, and decompose complex tasks. However, current coding benchmarks for Large Language Models (LLMs) primarily treat programming as a static, one-shot translation task. This "instruction-to-code" approach often leads to error propagation and a disconnect between generated code and its final execution environment. To address this, we introduce InterCode, a lightweight and flexible framework that formalizes interactive coding as a Reinforcement Learning (RL) environment. By providing safe, reproducible execution via Docker, InterCode enables models to use execution feedback as observations. We demonstrate that interaction significantly boosts performance—for instance, GPT-4's success rate on SQL tasks jumps from 9% to 74%—and provide a challenging new benchmark for advancing code understanding.

Introduction

Traditional AI coding benchmarks like HumanEval or MBPP evaluate models on their ability to generate a complete code block from a single prompt. While these benchmarks have spurred progress, they fail to capture the iterative nature of real-world software development. When humans program, we rarely write perfect code on the first attempt; instead, we engage in a "write-execute-test" loop, using error messages and output to refine our logic.

Static benchmarks create a "one-shot" bottleneck where a single typo or a misunderstood database schema results in total failure. InterCode bridges this gap by providing a standardized environment where AI agents can execute code, observe the results, and adjust their strategy dynamically.

The InterCode Framework

InterCode is designed to be language- and platform-agnostic, serving as a standard RL environment for code generation. It formalizes the coding process as a Partially Observable Markov Decision Process (POMDP):

  • Actions: The code snippets or commands generated by the agent.
  • Observations: The execution feedback, including standard output (stdout), error messages (stderr), or changes to the file system.
  • Environment: Each task runs within a self-contained Docker container, ensuring that execution is safe, isolated, and reproducible across different systems.

This framework is "plug-and-play," allowing researchers to wrap existing datasets into interactive environments or develop entirely new tasks. It is compatible with traditional sequence-to-sequence prompting as well as advanced agentic strategies.

Benchmark Environments and Datasets

We utilized InterCode to transform several popular static datasets into interactive benchmarks and introduced a new multi-step challenge:

  1. InterCode-Bash: Based on NL2Bash, featuring 200 tasks across four distinct file systems. Agents must navigate directories, manipulate files, and search system logs.
  2. InterCode-SQL: Derived from the Spider dataset, containing over 1,000 tasks across 20 databases. This environment tests the agent's ability to discover schemas and query complex data structures.
  3. InterCode-Python: Adapted from MBPP, focusing on function synthesis where the agent receives unit test feedback to refine its implementation.
  4. InterCode-CTF: A new dataset based on picoCTF (Capture the Flag) puzzles. These tasks are inherently multi-step and often require the agent to use multiple programming languages and tools to find a hidden "flag."

Experimental Evaluation

We evaluated state-of-the-art LLMs, including GPT-3.5 and GPT-4, using various prompting strategies to determine the impact of interaction.

The Power of Interaction

The most significant finding was the massive performance gain enabled by simple iterative feedback. In a "Single Turn" (static) setting, GPT-4 achieved only a 9.1% success rate on SQL tasks. When allowed to "Try Again" based on execution feedback, its success rate soared to 73.7%. Similar improvements were seen in Bash, where performance rose from 34.0% to 48.5%.

Prompting Strategies

We compared different reasoning frameworks, such as ReAct (Reason + Act) and Plan & Solve. Our results show that adaptive strategies like ReAct, which allow the model to adjust its plan based on real-time feedback, significantly outperform rigid planning. For example, GPT-3.5 using ReAct on SQL tasks achieved a 58.7% success rate, compared to 47.3% with a simple "Try Again" loop and only 10.5% in a static setting.

Key Insights

  • Context Discovery vs. Planning: Different tasks demand different interactive behaviors. SQL tasks are primarily about context discovery (e.g., querying table headers to understand the schema), while Bash tasks require modular planning and command piping.
  • The Plateau Effect: Interaction is not a panacea. Models eventually hit a plateau where they struggle to manage long context windows. In later turns, models often become distracted by previous failed attempts, leading to repetitive errors.
  • Human-AI Gap: Despite GPT-4's gains, a human baseline study on SQL tasks reached an 87% success rate, indicating that human-like modular problem-solving and error recovery still exceed current model capabilities.

Conclusion and Future Work

InterCode provides a robust foundation for the next generation of coding AI, moving beyond static generation toward interactive problem-solving. While the framework currently supports Bash, SQL, and Python, future work will expand to compiled languages like C++ and Java. Additionally, improving how models manage long-term interactive memory and expanding the complexity of multi-step tasks like CTF will be critical for closing the gap between AI and human developers.

Summary updated

Abstract

Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-to-code sequence transduction process, which has the potential for error propagation and a disconnect between the generated code and its final execution environment. To address this gap, we introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning (RL) environment, with code as actions and execution feedback as observations. Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution, and is compatible out-of-the-box with traditional seq2seq coding methods, while enabling the development of new methods for interactive code generation. We use InterCode to create three interactive code environments with Bash, SQL, and Python as action spaces, leveraging data from the static NL2Bash, Spider, and MBPP datasets. We demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct and Plan & Solve. Our results showcase the benefits of interactive code generation and demonstrate that InterCode can serve as a challenging benchmark for advancing code understanding and generation capabilities. InterCode is designed to be easily extensible and can even be used to create new tasks such as Capture the Flag, a popular coding puzzle that is inherently multi-step and involves multiple programming languages. Project site with code and data: https://intercode-benchmark.github.io
BibTeX
@inproceedings{yang2023intercode,
  author = {John Yang and Akshara Prabhakar and Karthik Narasimhan and Shunyu Yao},
  title = {{InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=iZSFixKS2Q},
}
Figure