Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation

Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao

Abstract

Large Language Models (LLMs) have become powerful tools for automated code generation. However, these models often overlook critical security practices, which can result in the generation of insecure code that contains vulnerabilities—weaknesses or flaws in the code that attackers can exploit to compromise a system. There has been limited exploration of strategies to guide LLMs in generating secure code and a lack of in-depth analysis of the effectiveness of LLMs in repairing code containing vulnerabilities. In this paper, we present a comprehensive evaluation of state-of-the-art LLMs by examining their inherent tendencies to produce insecure code, their capability to generate secure code when guided by self-generated vulnerability hints, and their effectiveness in repairing vulnerabilities when provided with different levels of feedback. Our study covers both proprietary and open-weight models across various scales and leverages established benchmarks to assess a wide range of vulnerability types. Through quantitative and qualitative analyses, we reveal that although LLMs are prone to generating insecure code, advanced models can benefit from vulnerability hints and fine-grained feedback to avoid or fix vulnerabilities. We also provide actionable suggestions to developers to reduce vulnerabilities when using LLMs for code generation.

Introduction

As AI-driven coding assistants like GitHub Copilot and ChatGPT become ubiquitous, a major red flag has emerged: they frequently generate insecure code. While previous research has highlighted these risks in massive proprietary models, there is a lack of comprehensive data on how smaller, open-weight models perform. This study addresses the challenge of vulnerability mitigation: how can we systematically guide an LLM to avoid security pitfalls during generation (proactive) and fix them after the fact (post-hoc) without requiring a human security expert to manually craft every prompt?

Methodology

The researchers evaluated LLMs across three distinct dimensions to understand and improve their security posture:

  1. Baseline Assessment: Establishing a baseline vulnerability rate by generating code using "vanilla" prompts (simple task descriptions).
  2. Proactive Prevention (Self-Generated Hints): A two-step process where the LLM is first asked to predict potential security risks (CWEs) for a given task. It then uses these self-generated hints to guide its own code generation.
  3. Post-Hoc Repair (Two Levels of Feedback): After generating vulnerable code, models were tasked with fixing it using feedback from CodeQL (a static analysis tool). This included "Direct Feedback" (raw error messages) and "Explained Feedback" (where GPT-4o explains the flaw and fix logic).
Overview of the three-pronged assessment framework for LLM-driven secure code generation.

Datasets

The study utilized two primary benchmarks for Python security:

  • SecurityEval: A dataset of 121 coding questions covering 69 different types of vulnerabilities (CWEs), derived from real-world scenarios.
  • SecCodePLT: A larger, synthesized dataset containing 1,071 questions covering 21 CWEs. It uses "mutated" versions of seed problems to test models across diverse contexts.

Results and Discussion

The empirical results reveal that vulnerability is a common trait across all tested models, with baseline vulnerability rates ranging from approximately 10% to over 40%.

Vulnerability rates (TarV-R and AllV-R) across different LLMs on SecCodePLT and SecurityEval datasets.

Key Insights

  • The "Double-Edged" Hint: Self-generated hints are powerful but risky. Precise hints drastically reduce vulnerabilities, but "hallucinated" or irrelevant hints can mislead models into writing more insecure code.
  • Contextualization is Key: Generic security definitions are less effective than "contextualized" hints that explain how a specific vulnerability might manifest in the current task.
  • The Reasoning Gap: Smaller models often fail to fix bugs when given raw error messages ("Direct Feedback"), as they lack the inherent reasoning to bridge the gap between identifying a flaw and implementing a logical fix.
  • The Power of Explanation: "Explained Feedback" acts as a force multiplier. By simplifying the task from "reasoning through a security flaw" to "following instructions," even mid-tier models can perform successful repairs.

Model Performance

GPT-4o leads the pack, showing the strongest ability to leverage hints and achieving a significant reduction in vulnerabilities. For instance, on the SecurityEval dataset, GPT-4o's vulnerability rate dropped from 41.3% to 13.2% after a single round of explained repair. Open-weight models like DeepSeek-Coder-V2-Lite (16B) and StarCoder2 (15B) also showed competitive performance, occasionally outperforming larger proprietary models in specific scenarios.

Limitations and Future Work

While promising, this study has limitations. It relies on static analysis (CodeQL), which may miss dynamic, runtime vulnerabilities. The evaluation was also limited to Python, and the single-turn interaction model does not fully capture the multi-turn nature of real-world developer-AI collaboration. Future research should explore multi-agent systems—where one AI acts as a "security reviewer" for another—and expand to more complex, multi-file software architectures.

Summary updated

Abstract

Large Language Models (LLMs) have become powerful tools for automated code generation. However, these models often overlook critical security practices, which can result in the generation of insecure code that contains vulnerabilities-weaknesses or flaws in the code that attackers can exploit to compromise a system. However, there has been limited exploration of strategies to guide LLMs in generating secure code and a lack of in-depth analysis of the effectiveness of LLMs in repairing code containing vulnerabilities. In this paper, we present a comprehensive evaluation of state-of-the-art LLMs by examining their inherent tendencies to produce insecure code, their capability to generate secure code when guided by self-generated vulnerability hints, and their effectiveness in repairing vulnerabilities when provided with different levels of feedback. Our study covers both proprietary and open-weight models across various scales and leverages established benchmarks to assess a wide range of vulnerability types. Through quantitative and qualitative analyses, we reveal that although LLMs are prone to generating insecure code, advanced models can benefit from vulnerability hints and fine-grained feedback to avoid or fix vulnerabilities. We also provide actionable suggestions to developers to reduce vulnerabilities when using LLMs for code generation.
BibTeX
@misc{yan2025guiding,
  author = {Hao Yan and Swapneel Suhas Vaidya and Xiaokuan Zhang and Ziyu Yao},
  title = {{Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation}},
  year = {2025},
  eprint = {2506.23034},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.23034},
}
Figure