Securing the Execution of Agent Generated Code

531 concepts346 papers2 proposals

The paradigm of AI-assisted programming has undergone a fundamental shift. Early applications treated Large Language Models (LLMs) as passive code autocompletes, relying on human developers to review, test, and integrate the suggested snippets. Today, however, language models are increasingly deployed as autonomous agents that write and immediately execute their own code.

Whether it is offloading complex math to a Python interpreter via Program of Thoughts prompting (Chen et al.), using autonomous data science agents to query databases (Maojun et al.), or building full-stack software applications via agentic platforms (Li), code has become the primary mechanism through which AI acts upon the digital world.

This shift introduces a severe security crisis. When an agent executes its own code, it eliminates human oversight—what Lee et al. call "collateral validation." If the generated code contains critical vulnerabilities or malicious logic, executing it can cause immediate, irreversible harm to the host system or database (Le et al.). Securing this loop requires a dual approach: mitigating the generation of vulnerable code and strictly isolating the execution environment through sandboxing.

Why Models Generate Insecure Code

We cannot rely solely on language models to write secure code by default. LLMs optimize for functional correctness and plausibility, not security invariants.

As Lyu et al. note, models replicate and amplify the security vulnerabilities present in their training data. Because they are trained on massive, unvetted public repositories, they inherit a vast legacy of software flaws (Du et al.). Furthermore, Jin demonstrates that models lack deep security reasoning, relying heavily on memorized patterns rather than a structural understanding of exploits and mitigations.

[Unvetted Public Code] ──> [LLM Training] ──> [Inherited Vulnerabilities]
                                                     │
                                                     ▼
[Global Context Lost]  <── [Incremental Diffs] <── [Functional Optimization]

This structural limitation is exacerbated by how agents interact with codebases. Models are highly prone to introducing vulnerabilities when generating small code modifications or "diffs" rather than writing full files from scratch. Chiang et al. argue that this localized focus causes the model to lose sight of global security invariants and broader architectural constraints. Compounding the issue, developers using AI assistants often experience a "false sense of security," making them less likely to catch these flaws manually (Srivastava et al.).

The Threat Landscape of Autonomous Agents

The risk escalates dramatically when code generation is paired with autonomous execution loops. If an agent is tasked with modifying a system or analyzing data, a single hallucinated or malicious command can compromise the environment.

For instance, small language models distilled for agent tasks often lack the capacity for complex safety steering, making them prone to generating harmful or irreversible system commands (Kang et al.). Similarly, data querying interfaces that dynamically generate executable database logic risk exposing underlying systems to catastrophic SQL injections or unintended data modification (Xu et al.).

This danger reaches its peak in self-referential agent architectures. Frameworks like Gödel Agent (Yin et al.) use runtime "monkey patching" to modify their own source code to optimize performance. In recursive self-improvement paradigms, models autonomously generate and evaluate successive iterations of themselves (Sugiyama et al.). Without absolute containment, an agent capable of modifying its own logic can easily bypass human-designed constraints, leading to a total loss of control over the host environment.

Sandboxing as the Primary Defense

Because post-execution feedback is inadequate for security-sensitive tasks—running a malicious script to see if it fails can already cause a data leak—the primary line of defense must be structural isolation. Sandboxing decouples the execution layer from the host system, ensuring that the blast radius of any generated code is strictly contained.

┌────────────────────────────────────────────────────────┐
│ Host System / Production Database                      │
│                                                        │
│   ┌────────────────────────────────────────────────┐   │
│   │ Isolated Sandbox Environment                   │   │
│   │                                                │   │
│   │   [LLM Agent] ──> Generates ──> [Untrusted]    │   │
│   │        ▲                         [Code]        │   │
│   │        │                            │          │   │
│   │        └─────── Returns ────────────▼          │   │
│   │                Safe Text/Visual Results        │   │
│   └────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────┘

Shi argues that executing model-generated code requires rigorous isolation to neutralize prompt injections and malicious logic. This sandboxing operates across multiple dimensions:

  • Environment Isolation: Generalist agent platforms like OpenHands (Li) execute all agent actions within secure, containerized environments. This ensures that actions like running bash commands or installing packages cannot alter the user's actual operating system. Similarly, advanced data analysis tools use offline execution environments to transform raw database schemas into structured text and graphical artifacts safely, preventing direct access to production databases (the authors).
  • Language-Subset Sandboxing: Beyond containerization, security can be enforced by restricting the programming language itself. Drawing from early web security frameworks, language-subset sandboxing limits the execution context by stripping out dangerous capabilities, such as direct file system access or dynamic code evaluation (eval) (Singh et al.). This restricts the agent to a safe mathematical or logical sandbox.

Complementary Safeguards: Alignment and Guardrails

While sandboxing physically contains the risks of execution, it is paired with inference-time techniques to improve the security of the generated code itself. Simple natural language reminders (e.g., "write secure code") often fail or cause the model to refuse the task entirely (Huang et al.). Instead, researchers use structured architectural interventions:

  • In-Context Safe Demonstrations: Rather than retraining models, frameworks like SecCoder (Zhang et al.) use a dense retriever to fetch a relevant, secure code demonstration at inference time, guiding the model's generation toward safe patterns.
  • Security Reasoning Traces: Forcing the model to think before it codes significantly mitigates risk. By prompting a model to generate a detailed security reasoning trace based on standardized vulnerability definitions before outputting the final code, the system activates latent security knowledge that is often missed during direct generation (Hasan et al.).
  • System-Level Guardrails: As outlined by Rebedea et al., securing agent applications requires treating the LLM as an integrated software component. This means wrapping the model in external verification layers—such as automated static analysis or runtime policy checkers—to enforce security constraints that the model cannot bypass on its own.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sandboxed_code_execution_aad5f492,
  title = {sandboxed code execution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sandboxed-code-execution/rdr_aad5f492648b4e0db65ee1bae84d45c3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_sandboxed_code_execution_aad5f492_v1,
  title = {sandboxed code execution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sandboxed-code-execution/rdr_aad5f492648b4e0db65ee1bae84d45c3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake