Introduction
As AI-assisted coding becomes ubiquitous—with industry leaders like Google reporting that 25% of new code is AI-generated—the security of this generated code has become a critical concern. While modern Large Language Models (LLMs) excel at functional programming, they frequently introduce vulnerabilities such as buffer overflows, SQL injections, and cross-site scripting (XSS). Current mitigation strategies, such as fine-tuning on secure datasets or simple "be secure" prompting, are either impractical for proprietary models or offer only marginal improvements.
SCGAgent addresses this gap by introducing an agentic workflow that guides standard LLMs through a structured "think-then-verify" cycle. This approach allows non-reasoning models to match or exceed the security performance of specialized reasoning models without sacrificing functional correctness.
Methodology: The SCGAgent Pipeline
SCGAgent operates as a multi-stage pipeline designed to mimic the interaction between a senior developer and a junior programmer. The workflow is structured to provide "scaffolded reasoning" through several key modules.
The process begins with Targeted Guideline Retrieval. Rather than providing a generic list of security rules, the agent prompts the LLM to predict specific Common Weakness Enumerations (CWEs) relevant to the task. It then retrieves expert-written guidelines tailored to those vulnerabilities.
To ensure that security enhancements do not break the code, the Enforce-Functionality Module generates a suite of unit tests alongside the initial code draft. A critical component of this workflow is the "Bug vs. Test" Decision logic: when a test fails, the agent evaluates whether the code is faulty or if the unit test itself is flawed, preventing the model from corrupting correct code to satisfy a broken test. Finally, the agent performs Iterative Refinement, applying security guidelines one by one and re-verifying functionality at each step.
Experimental Setup
The framework was evaluated using CWEval, a benchmark comprising 119 coding tasks across 31 different CWEs. The study focused on the C programming language, chosen for its manual memory management which presents a high-stakes environment for security vulnerabilities. Performance was measured using the Func-Sec@1 metric, which requires code to be both functionally correct and secure.
Key Results
The implementation of SCGAgent yielded significant improvements across multiple LLM architectures. Most notably, the workflow improved the security of Claude 3.7 Sonnet by approximately 25%, raising its Func-Sec@1 score from a 61% baseline to 76%.
Key findings include:
- Reasoning Parity: With the SCGAgent scaffold, the non-reasoning Sonnet 3.7 model achieved a 76% security score, effectively matching the 75% achieved by OpenAI’s o4-mini, a model with native reasoning capabilities.
- Functionality Retention: SCGAgent maintained a 98% functionality retention rate, avoiding the common pitfall where security-hardening techniques break the underlying program logic.
- Vulnerability Mitigation: In specific high-risk categories like Cross-Site Scripting (CWE-79), the workflow improved success rates from 0% to 100%.
Discussion and Limitations
While SCGAgent significantly boosts standard LLMs, the study observed a "Reasoning Model Mismatch." Native reasoning models like o4-mini actually saw a performance decrease when using the current workflow. This is attributed to the lower quality of unit tests currently produced by reasoning models, which can trigger a "garbage in, garbage out" cycle during the refinement phase.
Future research will focus on expanding the framework to higher-level languages like Python and Java, and moving toward fully autonomous agents that can dynamically decide when to invoke specific submodules rather than following a fixed control flow.
Summary updated