Executable Code as an Action Space

334 concepts203 papers3 proposals

When designing autonomous agents powered by Large Language Models (LLMs), the traditional approach treats the model as a step-by-step micro-manager. In a standard "Reasoning and Acting" (ReAct) loop, the agent observes the environment, generates a text or JSON string representing a single discrete action, waits for the environment's response, and repeats.

While intuitive, this token-by-token interaction model introduces significant bottlenecks. It is computationally expensive, introduces high latency due to frequent autoregressive generation, and struggles with long-horizon tasks that require complex logic like loops, conditional branching, or error recovery.

To overcome these limitations, a powerful paradigm shift has emerged: treating executable code as the primary action space. Instead of outputting static text commands, the agent synthesizes entire scripts (typically in Python) that are executed directly by a local interpreter or environment API. This approach shifts the burden of repetitive execution and local state management from the stochastic language model to a deterministic runtime environment.

Traditional Agent Loop:
[LLM] --(Single Action)--> [Environment] --(Observation)--> [LLM] (Repeat N times)

Programmatic Agent Loop:
[LLM] --(Complete Script with Loops/Logic)--> [Interpreter] <--> [Environment]
                                                     |
                                            (Executes Deterministically)

As shown by Wang et al. with the CodeAct framework, unifying an agent's reasoning and tool-use into a single programming interface drastically reduces the number of required communication turns. By allowing the model to write code that handles its own control flow, agents can execute complex, multi-step procedures within a single interaction turn.


Why It Works: Control Flow and Local State

The core reason programmatic action spaces outperform discrete action spaces lies in the expressive power of programming languages. Traditional reinforcement learning and agent frameworks struggle to represent complex conditional logic natively. If an agent needs to retry an action until a specific sensor condition is met, a traditional agent must query the LLM for every single attempt.

In contrast, representing actions as executable programs allows the agent to leverage standard programming constructs like while loops, if-else conditionals, and try-catch blocks. This insight dates back to classical planning architectures; as noted by Nau et al., embedding control flow into execution methods allows agents to handle environmental uncertainty and repetitive tasks far more robustly than static action sequences.

This architectural shift yields massive practical advantages in efficiency and latency:

  • Token and Cost Savings: Querying an LLM at every environmental step consumes a vast number of tokens. By framing planning as the generation of executable code scripts, a single model generation can govern multiple environment interactions. Tian et al. demonstrated that this programmatic framing can slash operational token costs by nearly 80% in complex simulations.
  • Latency Optimization: Bypassing the autoregressive generation loop for low-level checks minimizes latency. Winston et al. took this a step further by introducing "Agent JIT Compilation," compiling natural language instructions into optimized code plans to achieve up to a 10.4× speedup in web automation tasks.
  • High-Frequency Execution: In scenarios like robotics, large language models are too slow for real-time control loops. By generating a standalone executable script that encapsulates coordinate calculations and task logic, the model acts as a local teacher policy, bridging the latency gap to provide real-time guidance without per-step inference (Werner et al.).

Grounding via Programmatic Interfaces

For an LLM to write valid code that interacts with the physical or digital world, its output must be strictly constrained to valid environment commands. This is achieved by exposing the environment's capabilities as an API or a set of functional primitives.

In situated robot task planning, Singh introduced ProgPrompt, a method that structures the agent's prompt itself as a Python script. By defining available robotic actions as imported functions and listing available objects, the model's code generation is naturally constrained to a predefined API, preventing unsupported actions.

# Example of a Programmatic Action Interface (ProgPrompt style)
from robot_api import grab, move_to, open_app

def execute_task():
    # The LLM generates the body of this function
    for item in ["mug", "spoon"]:
        if item in detect_objects():
            move_to(item)
            grab(item)

This abstraction layer decouples high-level strategic reasoning from low-level execution mechanics. For example:

  • Robotics: In two-armed manipulation, Varley et al. used a programmatic representation to translate user instructions into scripts that call pre-defined robotic skills, enabling zero-shot composition of long-horizon behaviors without end-to-end neural torque control.
  • Operating Systems: Instead of predicting raw pixel coordinates to click on a screen—which is highly brittle—the OSCAR framework developed by Wang et al. generates Python code to interact with semantic element IDs, making the execution steps easier to verify and debug.
  • Autonomous Driving: Ma et al. decoupled high-level driving maneuvers (like overtaking) from low-level vehicle control by using an executable code interface, allowing a single programmatic output to govern complex maneuvers over several seconds.

Compositionality and Evolving Skill Libraries

A profound consequence of using code as an action space is that code is inherently composable and persistent. If an agent discovers a successful sequence of actions to solve a novel problem, it does not just discard that experience; it can save that code snippet as a new function.

This allows agents to build and maintain an evolving skill library. In complex open-world environments like Minecraft, the STEVE agent developed by Chai et al. avoids predicting raw motor commands by generating queries to retrieve and execute pre-defined code snippets from a modular skill library. Similarly, in multi-agent environments like StarCraft II, Zhao et al. showed that maintaining an evolving library of executable scripts allows agents to dynamically expand their behaviors and coordinate complex strategies.

As surveyed by Wu, this paradigm shifts the field away from frozen neural weights toward systems that write their own skills. Instead of operating within a rigid, pre-defined action space, frameworks like LearnAct (Zhao et al.) allow agents to iteratively create, refine, and store their own action space in the form of reusable Python functions.

[New Task] --> [Check Skill Library] --> [Compose Existing Code Functions]
                                                    |
                                           (If missing, synthesize
                                            & save new function)

This symbolic compositionality even extends to how agents model the world itself. Rather than trying to approximate environment transitions through high-dimensional neural feature vectors, Ahmed et al. demonstrated that agents can synthesize executable code to represent environment mechanics. This programmatic world modeling allows for explicit hypothesis testing and human-readable logic that can be instantly reused across different scenarios without retraining.

By treating code as the universal interface for action, planning, and world modeling, modern AI agents move closer to the flexibility of human software engineers—solving problems by writing reusable, deterministic programs rather than guessing one step at a time.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_code_based_action_representation_c78ed8b3,
  title = {code-based action representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/code-based-action-representation/rdr_c78ed8b35b414fc0bd85728e4b9a1fdd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_code_based_action_representation_c78ed8b3_v1,
  title = {code-based action representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/code-based-action-representation/rdr_c78ed8b35b414fc0bd85728e4b9a1fdd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake