Write Your Own CodeChecker: An Automated Test-Driven Checker Development Approach with LLMs

Yuanyuan Xie, Jun S. Liu, Jiwei Yan, Jinhao Huang, Jun Yan, Jian Zhang

TL;DR

AutoChecker is a new framework that uses Large Language Models (LLMs) to automatically write custom static code analysis rules. By using an iterative, test-driven development cycle and a specialized "logic-guided" API retrieval system, it achieves an 82.28% test pass rate, significantly outperforming standard LLM prompting methods.

What Problem Does This Paper Address?

Static analysis tools like PMD or SonarQube are essential for code quality, but their "out-of-the-box" rules often aren't enough for project-specific needs. While these tools allow developers to write custom checkers, very few actually do.

The barrier to entry is massive: checking frameworks are huge (often exceeding 30,000 lines of code), documentation is frequently incomplete, and the logic required to traverse an Abstract Syntax Tree (AST) is non-trivial. When developers try to use LLMs to bridge this gap, they hit two walls:

  1. Information Overload: Providing a full test suite at once overwhelms the LLM, leading to incomplete logic.
  2. API Hallucination: High-level rule descriptions don't map cleanly to specific framework APIs, causing LLMs to "guess" and invent non-existent functions.

Methods

The authors introduce AutoChecker, which treats checker development as an iterative process rather than a one-shot generation task. It relies on two core innovations:

  • Test-Driven Checker Development (TDCD): Instead of trying to pass all tests at once, AutoChecker picks one test case, generates a checker, validates it against the full suite, and then iteratively refines the code to fix failures or regressions. This "case-by-case" approach mimics how a human developer works and prevents the LLM from being overwhelmed.
  • Logic-guided API-context Retrieval: To stop the LLM from hallucinating APIs, AutoChecker first decomposes the rule into a "checking skeleton" of sub-operations. It then retrieves relevant API documentation from two specialized databases:
  • Full-API DB: A raw collection of over 11,000 PMD framework APIs.
  • Meta-API DB: A curated set of 354 "Meta-Operations" (like "Get name of class" or "Check if method is private") that act as a bridge between high-level intent and low-level code.

Datasets

The researchers evaluated AutoChecker using:

  • PMD Benchmark: 20 rules randomly selected from the official PMD ruleset (10 "easy" and 10 "hard" based on implementation complexity).
  • Real-world Projects: Five massive Java repositories, including Elasticsearch, Guava, and Spring Boot, were used to test if the generated checkers could actually find bugs in production-grade code.
  • API Databases: They constructed a comprehensive database of 11,641 framework-specific APIs from the PMD source code.

Key Insights and Ideas

  • Granularity Matters: The biggest hurdle for RAG (Retrieval-Augmented Generation) in coding is the "granularity mismatch." A user says "find static fields," but the API might be field.isStatic(). By breaking the rule into "Meta-Ops" first, the retrieval system finds the right tools for the job much more accurately.
  • The Power of the AST: AutoChecker doesn't just give the LLM the source code of a test case; it provides the parsed AST. By augmenting node names (e.g., ASTMethodDeclaration(length)), it helps the LLM "see" the connection between the code and the tree structure it needs to navigate.
  • Iterative Refinement is Key: One-shot generation (the AllCasesLLM baseline) performed poorly because the LLM couldn't maintain global logic while handling dozens of edge cases. Solving for one test at a time allowed the logic to grow organically and robustly.

Key Results

  • Superior Performance: AutoChecker achieved an average test pass rate of 82.28%. For comparison, the best baseline (using both RAG and Chain-of-Thought) only reached 38.93%.
  • Model Agnostic: While GPT-4 performed best, the system also worked remarkably well with DeepSeek-V3 (80.86% pass rate) and the self-hosted Qwen2.5-Coder-32B (79.01%).
  • Cost-Effective: Developing a rule with DeepSeek-V3 cost roughly $0.035 and took about 70 minutes. This is a fraction of the time and salary cost required for a senior engineer to learn the framework and write the rule manually.
  • Real-world Utility: Checkers generated by the tool were able to match the performance of official PMD checkers on large-scale projects after minor, LLM-assisted bug fixes.

Limitations and Future Work

The current version of AutoChecker is specifically designed for AST-based checkers, which look at code structure. It does not yet support flow-based checkers (which track data and control flow), as these are significantly more complex.

Additionally, while the approach is general, the current implementation is tied to the PMD framework and Java. The authors suggest that the "Meta-Op Set" is a step toward a framework-independent standard, but migrating to other tools (like CodeQL or ESLint) would require building new API databases. Future work aims to automate this database construction further to make the tool truly "plug-and-play" for any language or framework.

Summary updated

Abstract

With the rising demand for code quality assurance, developers are not only utilizing existing static code checkers but also seeking custom checkers to satisfy their specific needs. Nowadays, various code-checking frameworks provide extensive checker customization interfaces to meet this need. However, both the abstract checking logic and the complex API usage of large-scale checker frameworks make this task challenging. To this end, automated code checker generation is anticipated to ease the burden of checker development. In this paper, we propose AutoChecker, an innovative LLM-powered approach that can write code checkers automatically based on only a rule description and a test suite. To achieve comprehensive checking logic, AutoChecker incrementally updates the checker's logic by focusing on solving one selected case each time. To obtain precise API knowledge, during each iteration, it leverages fine-grained logic-guided API-context retrieval, where it first decomposes the checking logic into a series of sub-operations and then retrieves checker-related API-contexts for each sub-operation. For evaluation, we apply AutoChecker, five baselines, and three ablation methods using multiple LLMs to generate checkers for 20 randomly selected PMD rules. Experimental results show that AutoChecker significantly outperforms others across all effectiveness metrics, with an average test pass rate of 82.28%. Additionally, the checkers generated by AutoChecker can be successfully applied to real-world projects, matching the performance of official checkers.
BibTeX
@misc{xie2024write,
  author = {Yuanyuan Xie and Jun S. Liu and Jiwei Yan and Jinhao Huang and Jun Yan and Jian Zhang},
  title = {{Write Your Own CodeChecker: An Automated Test-Driven Checker Development Approach with LLMs}},
  year = {2024},
  eprint = {2411.06796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.06796},
}
Figure