Traditional software security relies heavily on static analysis—scanning source code for patterns that look like known vulnerabilities. However, as Large Language Models (LLMs) begin to autonomously generate complex backend systems and smart contracts, static analysis is reaching its breaking point. A pattern that looks like a vulnerability might be harmless in context, while a seemingly benign snippet might be catastrophic when executed.
The research direction emerging across these papers is a shift toward dynamic execution-based verification. Instead of asking "Does this code look dangerous?", researchers are building frameworks that ask "Can I actually exploit this code in a sandbox?"
The Gap Between Code and Reality
The fundamental problem with static security evaluation is the "exploitability gap." As Hossein Hajipour and colleagues point out in the CodeLMSec Benchmark, identifying a potential weakness is not the same as confirming a risk. In real-world environments, system configurations, library versions, and runtime guards can render a "vulnerable" pattern unreachable.
This gap leads to high false-positive rates, which Chao Guo and Chengquan Guo argue is a primary friction point for AI-assisted development. Their work on BlueCodeAgent demonstrates that by integrating dynamic execution, a system can "overturn" initial security flags if the code proves to be safe under stress, improving developer trust.
The Sandbox as a Truth Oracle
To move beyond static "vibes," researchers are employing isolated runtime environments—typically Docker containers or specialized virtual machines—to serve as a "truth oracle."
In JAWS-BENCH, Shoumik Saha and team use live execution to uncover "agentic amplification." They found that when an LLM is wrapped in an agentic framework, it is 1.6x more likely to bypass its own safety training because it can reason through multi-step execution paths that static filters miss. Similarly, Mark Vero and Niels Mündler developed BAXBENCH to test full backend applications. They found that nearly half of the programs deemed "correct" by static evaluators actually contained critical, exploitable vulnerabilities when subjected to live attacks.
Automated Proof of Concept Generation
If the goal is to prove exploitability, the system must be able to write its own exploits. This has led to the rise of Automated Exploit Generation (AEG).
Frameworks like SEC-bench by Hwiwon Lee and LINGMING ZHANG use a dual-stage validation loop. An agent must first generate a functional Proof-of-Concept (PoC) exploit that triggers a failure (like a memory sanitizer crash) before it is allowed to attempt a fix. This ensures the agent is solving a real, reproducible bug rather than hallucinating a security flaw.
In the domain of decentralized finance, Zhen Xiao and Qin Wang introduced Prompt to Pwn, which uses LLMs to generate smart contract exploits. These scripts are executed on private blockchain replicas to monitor for "invariant violations"—such as an unauthorized increase in an attacker's balance—providing definitive proof of a "rug pull" vulnerability.
Contrastive and Invariant Verification
How do we know an exploit succeeded because of a specific vulnerability and not just a random crash? Researchers use two primary techniques:
- Differential/Contrastive Testing: AutoBaxBuilder, developed by Tobias von Arx and Maximilian Baader, validates exploits by running them against two versions of the code: a "weakened" version (with the bug) and a "hardened" version (with the patch). A valid exploit must succeed on the first and fail on the second.
- Semantic Invariants: Instead of looking for crashes, Haijun Wang and Shang-Wei Lin use ContraMaster to monitor "semantic invariants"—rules about how a program should behave. If a sequence of transactions causes a contract to lose more money than it should, the vulnerability is confirmed, regardless of whether the code "looks" wrong.
The Future: From Detection to Action
The shift toward dynamic verification is turning security from a game of "guessing" into a science of "proving." As Saad Ullah and Wenbo Guo demonstrate with CVE-GENIE, we can now automate the reproduction of real-world CVEs for just a few dollars per vulnerability. By prioritizing empirical exploitability over static patterns, this research direction provides a path toward AI agents that don't just write code, but can prove that the code they write is resilient to the very attacks they are capable of generating.