While Large Language Models (LLMs) have revolutionized natural language processing and general software development, their utility in the high-stakes world of system security remains a frontier. Security tasks require more than just syntactic understanding; they demand deep reasoning about control flows, data dependencies, and low-level machine instructions. This research provides a comprehensive evaluation of ChatGPT (GPT-3.5 and GPT-4) across seven critical software security domains, identifying both the transformative potential and the current limitations of these models.
The Research Scope
The study stress-tests LLMs against a broad spectrum of "hard" security tasks. These include vulnerability repair, bug fixing, patching, root cause analysis, decompilation, symbolic execution, and fuzzing. By moving beyond simple code generation, the researchers aimed to see if ChatGPT could function as a reliable assistant for security engineers.
Methodology and Datasets
The researchers adopted a systematic "prompt-and-evaluate" methodology, utilizing a mix of synthetic and real-world data:
- Synthetic Vulnerabilities: 100 cases from the SARD dataset covering 10 common CWEs.
- Real-World Vulnerabilities: 34 unique CVEs from open-source projects.
- Bug Fixing: The QuixBugs benchmark (40 Python and 40 Java programs).
- Decompilation: 30 LeetCode solutions compiled into x86_64 assembly.
- Vulnerability Repair: 154 synthetic cases and 12 real-world OpenSSL CVEs.
Evaluation involved quantitative benchmarking (measuring precision and recall) and qualitative case studies to observe where model logic succeeds or breaks down.
Evolution of Capability: GPT-3.5 vs. GPT-4
A central finding of the study is the massive leap in reasoning capability from GPT-3.5 to GPT-4. While GPT-3.5 often fails to identify subtle vulnerabilities or provides generic "as an AI language model" refusals, GPT-4 demonstrates a much higher success rate in pinpointing specific security flaws.
Key Insights and Results
1. Semantic Decompilation
Traditional decompilers like IDA Pro are effective at recovering code structure but often produce unreadable variable names (e.g., v1, v2). GPT-4 acts as a "semantic" decompiler, reading assembly logic to infer meaningful names (like root, left, right for tree traversals). In functional correctness, GPT-4 achieved 53.3%, significantly outperforming IDA Pro’s 33.3%.
2. Reasoning Over Pattern Matching
Unlike static analysis tools, ChatGPT can perform Root Cause Analysis. When provided with a stack trace, it can explain the causal chain leading to a crash rather than just pointing to a line of code.
3. Quantitative Performance
- Vulnerability Detection: GPT-4 achieved 98% recall on synthetic code, though this dropped to 50% on complex real-world CVEs.
- Bug Fixing: GPT-4 successfully fixed 85% of Python bugs and 82.5% of Java bugs in QuixBugs, compared to 60% and 35% for GPT-3.5.
- Vulnerability Repair: GPT-4 solved 95% of synthetic repair challenges.
Limitations and the "Context Wall"
Despite these gains, three primary bottlenecks remain:
- Context Window: Models struggle when vulnerabilities are buried in large, multi-file projects with complex dependencies.
- Numerical Reasoning: ChatGPT often fails at precise math, such as miscalculating byte offsets even when it understands the conceptual vulnerability.
- Open-Loop Nature: Current tasks like fuzzing are "open-loop." Future systems would benefit from a "closed-loop" approach where the LLM interacts with a debugger or compiler to refine its output based on real-time feedback.
In conclusion, while ChatGPT is not yet ready to replace security engineers, it has become a powerful force multiplier for code auditing and reverse engineering.
Summary updated