Bug Finding With Agentic Symbolic Execution

George Granberry, Wolfgang Ahrendt, Andy Gordon, Moa JohanssonarXiv 2026 · arXiv

TL;DR

This paper introduces an "agentic harness" that uses Large Language Models (LLMs) to automate the two biggest hurdles in symbolic execution: writing formal code contracts and modeling complex external environments like databases. By pairing the symbolic execution tool CrossHair with an AI agent, the authors successfully analyzed over 1.1 million lines of production Python code, finding 1,050 confirmed bugs that pure LLM inspection often misses.

What Problem Does This Paper Address?

Symbolic execution is a powerful formal method that explores program paths by treating inputs as mathematical symbols rather than concrete values. It’s legendary for finding "impossible" edge-case bugs. However, it rarely makes it out of the lab into large-scale web applications for two reasons:

  1. The Oracle Problem: You need to tell the tool what "correct" behavior looks like using formal contracts (pre- and post-conditions). Most real-world code has zero contracts.
  2. The Environment Problem: Modern apps are "hollow"—their logic depends heavily on external systems like databases (ORMs). A symbolic engine crashes the moment it hits a database call it doesn't understand.

Manually writing stubs and contracts for a million-line repository is a non-starter for most engineering teams. This paper asks: Can we use LLM agents to do the "boring" manual labor of formal methods?

Methods

The authors developed a pipeline called crosshair-bugs that follows a core design principle: The LLM proposes, the solver checks, and concrete execution confirms.

The system operates in five automated stages:

  1. Environment Synthesis: The agent inspects the repository to understand the database schema and framework (e.g., Django or SQLAlchemy). it then synthesizes "stubs" (mocks) and "lazy initialization" logic so CrossHair can execute code that touches the database without crashing.
  2. Contract Generation: The agent reads function bodies and writes formal Python docstring contracts (e.g., post: _ >= 0).
  3. Symbolic Execution: CrossHair runs on the instrumented code, using an SMT solver (Z3) to find inputs that violate the agent-written contracts.
  4. Audit: Because the synthesized environment is an "over-approximation" (it might allow states a real database wouldn't), a separate audit agent filters out false positives.
  5. Witness Validation: For every suspected bug, the system generates a concrete Python test case and runs it against the unmodified application to prove the bug is real.
Fig. 1. System overview.

Datasets

The authors evaluated their tool on 1,156,044 lines of Python code across 11 popular open-source web applications, including:

  • Zulip (Team chat)
  • NetBox (Infrastructure modeling)
  • Apache Superset (Data exploration)
  • Paperless-ngx (Document management)

They compared their results against a baseline of "direct LLM inspection," where the same model (Claude 3.5 Opus) was simply asked to find bugs by reading the code.

Key Insights and Ideas

  • Reach is Everything: Without environment modeling, symbolic execution is blind. By synthesizing ORM stubs, the authors increased the "reach" (the percentage of functions that could actually be executed) from a measly 4.5% to 81%.
  • Complementary Strengths: LLMs are great at finding "logical" bugs (e.g., "this calculation is wrong"). Symbolic execution is superior at finding "input-domain" bugs—corner cases like NUL bytes, empty strings, or unusual Unicode characters that a human (or an LLM) would likely assume are "out of scope."
  • The "Agentic Harness" as a Force Multiplier: The LLM isn't replacing the formal tool; it's acting as the "glue" that makes the tool applicable to messy, real-world software.

Key Results

  • Bug Yield: The pipeline found 1,050 witness-reproduced bugs. 221 of these were found specifically by the symbolic execution engine, while 829 were flagged by the agent during the contract-writing phase.
  • Reach Improvement: As shown in Figure 4, the full synthesized environment (dark green) drastically outperforms bare symbolic execution (light green) across all repositories.
Fig. 4. Executed contracted functions per repository under three environments: bare CrossHair, framework initialization only, and full environment synthesis (crosshair-bugs). Pooled over 5,406 functions: 4.5%, 29.5% and 81%.
  • Exploration Depth: While symbolic execution reached 81% of functions, it only "exhausted" (fully explored) 3.6% of them due to the classic problem of path explosion. However, 28.6% were explored to a "bounded" depth of at least 25 paths, providing significantly higher assurance than a standard unit test.
Fig. 5. Exploration tier of each contracted function, per repository and pooled (n=5,406). Exhausted: all paths explored; bounded: at least 25 checked paths; shallow: 1–24; refuted: a condition was violated (candidate finding); blocked: no checked path.
  • Beating the Baseline: The pipeline found 312 bugs that the pure LLM inspection baseline missed. These were often subtle crashes, such as a URL parser failing on a malformed string like //[ or a password strength checker crashing on an unexpectedly long input.

Limitations and Future Work

  • Cost: Running this pipeline is expensive. The LLM costs for the study were over $2,800 for the 11 repositories.
  • Path Explosion: The tool still struggles with "classic" symbolic execution bottlenecks like complex string manipulations, date parsing, and C extensions.
  • Stateful Databases: The current environment model is "stateless"—it models what a single query might return but doesn't perfectly track how one write operation affects a subsequent read.
  • Future Directions: The authors suggest that the same "agentic" approach could be used to model other boundaries, such as HTTP APIs, file systems, or C-based Python extensions.

Summary updated

Abstract

Symbolic execution, in tools such as CrossHair for Python, is a powerful technique for finding bugs, but two barriers hinder its adoption: the human effort of annotating code with assertions, and the need to model the execution environment outside Python, such as an external database. To overcome both, we design an agentic harness that orchestrates the symbolic execution tool CrossHair together with an AI coding agent. Without human intervention, it instruments a repository with assertions, synthesizes a constraint-aware symbolic environment for the database, runs symbolic execution, and constructs a concrete test as evidence for each bug it reports. We evaluate our harness on 1,156,044 lines of Python code over 11 popular open-source Python web applications (Zulip, NetBox, Apache Superset, and others). With the synthesized environment, CrossHair executes 81% of the 5,406 contracted functions, against 4.5% without it. Our pipeline finds 1,050 bugs, each reproduced by a test against the unmodified application, 221 of them by CrossHair, and 312 of them are missed by a baseline in which the same LLM inspects the code. The two find different kinds of bugs: CrossHair's are mostly crashes on input-domain corner cases that validation admits, such as empty strings, missing fields and unusual characters, while inspection's are mostly bugs that produce wrong output. Our framework thus lets symbolic execution supplement inspection-based bug finding.
BibTeX
@misc{granberry2026bug,
  author = {George Granberry and Wolfgang Ahrendt and Andy Gordon and Moa Johansson},
  title = {{Bug Finding With Agentic Symbolic Execution}},
  year = {2026},
  eprint = {2611.15596},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15596},
}
Figure