TL;DR
This paper introduces a novel "context-driven" prompting technique to significantly improve GPT-4's ability to audit smart contracts for vulnerabilities. By strategically chunking code and providing targeted context, their method achieves a 96% detection rate, far surpassing standard prompting methods, making AI-assisted smart contract auditing more effective and reliable.
What Problem Does This Paper Address?
Smart contracts are the backbone of many blockchain applications, promising efficiency and transparency. However, their immutable nature means vulnerabilities can lead to catastrophic financial losses. Manual auditing is crucial but incredibly time-consuming and requires deep expertise. While Large Language Models (LLMs) like GPT-4 show promise in assisting auditors (co-auditing), their effectiveness is hampered by the sheer volume and complexity of smart contract code. LLMs struggle with long code contexts, leading to missed vulnerabilities. This paper tackles the challenge of making LLMs more effective co-auditors by developing smarter ways to present smart contract code and audit goals to them.
Methods
The core innovation is a "context-driven prompting technique" that employs three main strategies:
- Code Scoping (CCL-based chunking): Instead of feeding the entire code or relying on LLM's default truncation, this method breaks down long code into smaller, self-contained "Code Call Lists" (CCLs). These CCLs are generated by first flattening the code (consolidating all imported contracts and libraries into a single file), then building a Function Call Graph (FCG) to understand inter-function dependencies, and finally creating a CCL for each function that includes the function itself, its callers, and its callees. This ensures relevant code segments are provided without overwhelming the LLM.
- Assessment Scoping (CAQ-based prompts): This involves tailoring prompts to focus on specific types of vulnerabilities. The authors use Common Audit Questions (CAQs) derived from existing audit reports to guide the LLM's search.
- Reporting Scoping (CWE-based prompts): This strategy aims to standardize the output format by leveraging the Common Weakness Enumeration (CWE) to categorize and report vulnerabilities.
Datasets
The study utilizes two publicly available datasets for evaluation:
- Smart Bugs Curated: A collection of 69 vulnerable smart contracts categorized by vulnerability type.
- SolidiFI-Benchmark: A repository of 50 vulnerable contracts covering various bug types.
To mitigate potential bias from LLMs being trained on existing datasets, the researchers also created Dataset B, which consists of unlabeled functions from real-world smart contracts, paired with corresponding Contract Contextual Labels (CCL).
Key Insights and Ideas
- Chunking is Crucial: The paper strongly demonstrates that breaking down large codebases into manageable, dependency-aware chunks (CCL) is far more effective than feeding the entire code to the LLM.
- Context is King: The way information is presented to the LLM dramatically impacts its performance. Simply providing more code isn't always better; providing the right code and context is key.
- CAQ Prompts Shine: While both CAQ and CWE prompts showed improvements with chunking, the CAQ-based prompts were found to be more efficient and capable of detecting a wider range of vulnerabilities.
- Human Validation Matters: Qualitative analysis by expert auditors confirmed that the proposed approach significantly improves GPT-4's ability to identify vulnerabilities, even on unlabeled datasets.
Key Results
- The proposed context-driven approach achieved a 96% detection rate for vulnerable functions, significantly outperforming the native prompting approach (53%).
- CCL-based chunking consistently outperformed full-code prompting across both CAQ and CWE prompt designs.
- Human auditors rated GPT-generated reports as correct in 56% of cases and relevant in 46% of instances, with these figures rising to 76.5% and 75% for partial correctness and relevance, respectively, when using the proposed approach.
- The CAQ+CCL approach outperformed traditional static analysis tools like Slither, Mythril, and Oyente in detecting vulnerabilities.
Limitations and Future Work
- Internal Threats: The automated CCL extraction script might erroneously select incorrect imported functions due to multiple possible implementations for the same function signature. The authors mitigated this by cross-examining CCLs. They also note that the absence of comments in some contracts could affect performance.
- External Threats: The use of publicly available datasets might mean GPT-4 was trained on them, potentially biasing results. Dataset B, with unlabeled functions, was created to address this. However, the possibility of overlaps with other datasets remains.
- Variable Ambiguity: GPT might struggle to distinguish between state variables with the same name in different contracts. Future work will involve prompting this declaration area.
- Future Work: The authors suggest exploring more advanced prompt engineering techniques and further investigating the impact of context definition and structure on LLM performance. They also aim to develop more robust methods for handling complex dependencies and ensuring the LLM can accurately distinguish between different contract variables.
Summary updated