Abstract
Humans write code in a fundamentally interactive manner, relying on constant execution feedback to correct errors, resolve ambiguities, and decompose complex tasks. However, current coding benchmarks for Large Language Models (LLMs) primarily treat programming as a static, one-shot translation task. This "instruction-to-code" approach often leads to error propagation and a disconnect between generated code and its final execution environment. To address this, we introduce InterCode, a lightweight and flexible framework that formalizes interactive coding as a Reinforcement Learning (RL) environment. By providing safe, reproducible execution via Docker, InterCode enables models to use execution feedback as observations. We demonstrate that interaction significantly boosts performance—for instance, GPT-4's success rate on SQL tasks jumps from 9% to 74%—and provide a challenging new benchmark for advancing code understanding.
Introduction
Traditional AI coding benchmarks like HumanEval or MBPP evaluate models on their ability to generate a complete code block from a single prompt. While these benchmarks have spurred progress, they fail to capture the iterative nature of real-world software development. When humans program, we rarely write perfect code on the first attempt; instead, we engage in a "write-execute-test" loop, using error messages and output to refine our logic.
Static benchmarks create a "one-shot" bottleneck where a single typo or a misunderstood database schema results in total failure. InterCode bridges this gap by providing a standardized environment where AI agents can execute code, observe the results, and adjust their strategy dynamically.
The InterCode Framework
InterCode is designed to be language- and platform-agnostic, serving as a standard RL environment for code generation. It formalizes the coding process as a Partially Observable Markov Decision Process (POMDP):
- Actions: The code snippets or commands generated by the agent.
- Observations: The execution feedback, including standard output (stdout), error messages (stderr), or changes to the file system.
- Environment: Each task runs within a self-contained Docker container, ensuring that execution is safe, isolated, and reproducible across different systems.
This framework is "plug-and-play," allowing researchers to wrap existing datasets into interactive environments or develop entirely new tasks. It is compatible with traditional sequence-to-sequence prompting as well as advanced agentic strategies.
Benchmark Environments and Datasets
We utilized InterCode to transform several popular static datasets into interactive benchmarks and introduced a new multi-step challenge:
- InterCode-Bash: Based on NL2Bash, featuring 200 tasks across four distinct file systems. Agents must navigate directories, manipulate files, and search system logs.
- InterCode-SQL: Derived from the Spider dataset, containing over 1,000 tasks across 20 databases. This environment tests the agent's ability to discover schemas and query complex data structures.
- InterCode-Python: Adapted from MBPP, focusing on function synthesis where the agent receives unit test feedback to refine its implementation.
- InterCode-CTF: A new dataset based on picoCTF (Capture the Flag) puzzles. These tasks are inherently multi-step and often require the agent to use multiple programming languages and tools to find a hidden "flag."
Experimental Evaluation
We evaluated state-of-the-art LLMs, including GPT-3.5 and GPT-4, using various prompting strategies to determine the impact of interaction.
The Power of Interaction
The most significant finding was the massive performance gain enabled by simple iterative feedback. In a "Single Turn" (static) setting, GPT-4 achieved only a 9.1% success rate on SQL tasks. When allowed to "Try Again" based on execution feedback, its success rate soared to 73.7%. Similar improvements were seen in Bash, where performance rose from 34.0% to 48.5%.
Prompting Strategies
We compared different reasoning frameworks, such as ReAct (Reason + Act) and Plan & Solve. Our results show that adaptive strategies like ReAct, which allow the model to adjust its plan based on real-time feedback, significantly outperform rigid planning. For example, GPT-3.5 using ReAct on SQL tasks achieved a 58.7% success rate, compared to 47.3% with a simple "Try Again" loop and only 10.5% in a static setting.
Key Insights
- Context Discovery vs. Planning: Different tasks demand different interactive behaviors. SQL tasks are primarily about context discovery (e.g., querying table headers to understand the schema), while Bash tasks require modular planning and command piping.
- The Plateau Effect: Interaction is not a panacea. Models eventually hit a plateau where they struggle to manage long context windows. In later turns, models often become distracted by previous failed attempts, leading to repetitive errors.
- Human-AI Gap: Despite GPT-4's gains, a human baseline study on SQL tasks reached an 87% success rate, indicating that human-like modular problem-solving and error recovery still exceed current model capabilities.
Conclusion and Future Work
InterCode provides a robust foundation for the next generation of coding AI, moving beyond static generation toward interactive problem-solving. While the framework currently supports Bash, SQL, and Python, future work will expand to compiled languages like C++ and Java. Additionally, improving how models manage long-term interactive memory and expanding the complexity of multi-step tasks like CTF will be critical for closing the gap between AI and human developers.
Summary updated