Exploring the Limits of ChatGPT in Software Security Applications

Fangzhou Wu, Qingzhao Zhang, Ati Priya Bajaj, Tiffany Bao, Ning Zhang, Ruoyu Wang, Chaowei XiaoarXiv 2023 · DOI · arXiv

While Large Language Models (LLMs) have revolutionized natural language processing and general software development, their utility in the high-stakes world of system security remains a frontier. Security tasks require more than just syntactic understanding; they demand deep reasoning about control flows, data dependencies, and low-level machine instructions. This research provides a comprehensive evaluation of ChatGPT (GPT-3.5 and GPT-4) across seven critical software security domains, identifying both the transformative potential and the current limitations of these models.

The Research Scope

The study stress-tests LLMs against a broad spectrum of "hard" security tasks. These include vulnerability repair, bug fixing, patching, root cause analysis, decompilation, symbolic execution, and fuzzing. By moving beyond simple code generation, the researchers aimed to see if ChatGPT could function as a reliable assistant for security engineers.

Taxonomy of ChatGPT applications in software security evaluated in the study.

Methodology and Datasets

The researchers adopted a systematic "prompt-and-evaluate" methodology, utilizing a mix of synthetic and real-world data:

  • Synthetic Vulnerabilities: 100 cases from the SARD dataset covering 10 common CWEs.
  • Real-World Vulnerabilities: 34 unique CVEs from open-source projects.
  • Bug Fixing: The QuixBugs benchmark (40 Python and 40 Java programs).
  • Decompilation: 30 LeetCode solutions compiled into x86_64 assembly.
  • Vulnerability Repair: 154 synthetic cases and 12 real-world OpenSSL CVEs.

Evaluation involved quantitative benchmarking (measuring precision and recall) and qualitative case studies to observe where model logic succeeds or breaks down.

Evolution of Capability: GPT-3.5 vs. GPT-4

A central finding of the study is the massive leap in reasoning capability from GPT-3.5 to GPT-4. While GPT-3.5 often fails to identify subtle vulnerabilities or provides generic "as an AI language model" refusals, GPT-4 demonstrates a much higher success rate in pinpointing specific security flaws.

A comparison showing GPT-4 successfully identifying a buffer overflow vulnerability that GPT-3.5 failed to detect.

Key Insights and Results

1. Semantic Decompilation

Traditional decompilers like IDA Pro are effective at recovering code structure but often produce unreadable variable names (e.g., v1, v2). GPT-4 acts as a "semantic" decompiler, reading assembly logic to infer meaningful names (like root, left, right for tree traversals). In functional correctness, GPT-4 achieved 53.3%, significantly outperforming IDA Pro’s 33.3%.

2. Reasoning Over Pattern Matching

Unlike static analysis tools, ChatGPT can perform Root Cause Analysis. When provided with a stack trace, it can explain the causal chain leading to a crash rather than just pointing to a line of code.

3. Quantitative Performance

  • Vulnerability Detection: GPT-4 achieved 98% recall on synthetic code, though this dropped to 50% on complex real-world CVEs.
  • Bug Fixing: GPT-4 successfully fixed 85% of Python bugs and 82.5% of Java bugs in QuixBugs, compared to 60% and 35% for GPT-3.5.
  • Vulnerability Repair: GPT-4 solved 95% of synthetic repair challenges.

Limitations and the "Context Wall"

Despite these gains, three primary bottlenecks remain:

  1. Context Window: Models struggle when vulnerabilities are buried in large, multi-file projects with complex dependencies.
  2. Numerical Reasoning: ChatGPT often fails at precise math, such as miscalculating byte offsets even when it understands the conceptual vulnerability.
  3. Open-Loop Nature: Current tasks like fuzzing are "open-loop." Future systems would benefit from a "closed-loop" approach where the LLM interacts with a debugger or compiler to refine its output based on real-time feedback.

In conclusion, while ChatGPT is not yet ready to replace security engineers, it has become a powerful force multiplier for code auditing and reverse engineering.

Summary updated

Abstract

Large language models (LLMs) have undergone rapid evolution and achieved remarkable results in recent times. OpenAI's ChatGPT, backed by GPT-3.5 or GPT-4, has gained instant popularity due to its strong capability across a wide range of tasks, including natural language tasks, coding, mathematics, and engaging conversations. However, the impacts and limits of such LLMs in system security domain are less explored. In this paper, we delve into the limits of LLMs (i.e., ChatGPT) in seven software security applications including vulnerability detection/repair, debugging, debloating, decompilation, patching, root cause analysis, symbolic execution, and fuzzing. Our exploration reveals that ChatGPT not only excels at generating code, which is the conventional application of language models, but also demonstrates strong capability in understanding user-provided commands in natural languages, reasoning about control and data flows within programs, generating complex data structures, and even decompiling assembly code. Notably, GPT-4 showcases significant improvements over GPT-3.5 in most security tasks. Also, certain limitations of ChatGPT in security-related tasks are identified, such as its constrained ability to process long code contexts.
BibTeX
@misc{wu2023exploring,
  author = {Fangzhou Wu and Qingzhao Zhang and Ati Priya Bajaj and Tiffany Bao and Ning Zhang and Ruoyu Wang and Chaowei Xiao},
  title = {{Exploring the Limits of ChatGPT in Software Security Applications}},
  year = {2023},
  eprint = {2312.05275},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.05275},
}
Figure