TL;DR
AutoChecker is a new framework that uses Large Language Models (LLMs) to automatically write custom static code analysis rules. By using an iterative, test-driven development cycle and a specialized "logic-guided" API retrieval system, it achieves an 82.28% test pass rate, significantly outperforming standard LLM prompting methods.
What Problem Does This Paper Address?
Static analysis tools like PMD or SonarQube are essential for code quality, but their "out-of-the-box" rules often aren't enough for project-specific needs. While these tools allow developers to write custom checkers, very few actually do.
The barrier to entry is massive: checking frameworks are huge (often exceeding 30,000 lines of code), documentation is frequently incomplete, and the logic required to traverse an Abstract Syntax Tree (AST) is non-trivial. When developers try to use LLMs to bridge this gap, they hit two walls:
- Information Overload: Providing a full test suite at once overwhelms the LLM, leading to incomplete logic.
- API Hallucination: High-level rule descriptions don't map cleanly to specific framework APIs, causing LLMs to "guess" and invent non-existent functions.
Methods
The authors introduce AutoChecker, which treats checker development as an iterative process rather than a one-shot generation task. It relies on two core innovations:
- Test-Driven Checker Development (TDCD): Instead of trying to pass all tests at once, AutoChecker picks one test case, generates a checker, validates it against the full suite, and then iteratively refines the code to fix failures or regressions. This "case-by-case" approach mimics how a human developer works and prevents the LLM from being overwhelmed.
- Logic-guided API-context Retrieval: To stop the LLM from hallucinating APIs, AutoChecker first decomposes the rule into a "checking skeleton" of sub-operations. It then retrieves relevant API documentation from two specialized databases:
- Full-API DB: A raw collection of over 11,000 PMD framework APIs.
- Meta-API DB: A curated set of 354 "Meta-Operations" (like "Get name of class" or "Check if method is private") that act as a bridge between high-level intent and low-level code.
Datasets
The researchers evaluated AutoChecker using:
- PMD Benchmark: 20 rules randomly selected from the official PMD ruleset (10 "easy" and 10 "hard" based on implementation complexity).
- Real-world Projects: Five massive Java repositories, including Elasticsearch, Guava, and Spring Boot, were used to test if the generated checkers could actually find bugs in production-grade code.
- API Databases: They constructed a comprehensive database of 11,641 framework-specific APIs from the PMD source code.
Key Insights and Ideas
- Granularity Matters: The biggest hurdle for RAG (Retrieval-Augmented Generation) in coding is the "granularity mismatch." A user says "find static fields," but the API might be
field.isStatic(). By breaking the rule into "Meta-Ops" first, the retrieval system finds the right tools for the job much more accurately. - The Power of the AST: AutoChecker doesn't just give the LLM the source code of a test case; it provides the parsed AST. By augmenting node names (e.g.,
ASTMethodDeclaration(length)), it helps the LLM "see" the connection between the code and the tree structure it needs to navigate. - Iterative Refinement is Key: One-shot generation (the
AllCasesLLMbaseline) performed poorly because the LLM couldn't maintain global logic while handling dozens of edge cases. Solving for one test at a time allowed the logic to grow organically and robustly.
Key Results
- Superior Performance: AutoChecker achieved an average test pass rate of 82.28%. For comparison, the best baseline (using both RAG and Chain-of-Thought) only reached 38.93%.
- Model Agnostic: While GPT-4 performed best, the system also worked remarkably well with DeepSeek-V3 (80.86% pass rate) and the self-hosted Qwen2.5-Coder-32B (79.01%).
- Cost-Effective: Developing a rule with DeepSeek-V3 cost roughly $0.035 and took about 70 minutes. This is a fraction of the time and salary cost required for a senior engineer to learn the framework and write the rule manually.
- Real-world Utility: Checkers generated by the tool were able to match the performance of official PMD checkers on large-scale projects after minor, LLM-assisted bug fixes.
Limitations and Future Work
The current version of AutoChecker is specifically designed for AST-based checkers, which look at code structure. It does not yet support flow-based checkers (which track data and control flow), as these are significantly more complex.
Additionally, while the approach is general, the current implementation is tied to the PMD framework and Java. The authors suggest that the "Meta-Op Set" is a step toward a framework-independent standard, but migrating to other tools (like CodeQL or ESLint) would require building new API databases. Future work aims to automate this database construction further to make the tool truly "plug-and-play" for any language or framework.
Summary updated