Automatically Write Code Checker: An LLM-based Approach with Logic-guided API Retrieval and Case by Case Iteration

Yuanyuan Xie, Jun Liu, Jiwei Yan, Jinhao Huang, Jun Yan, Jian Zhang

Abstract

Static analysis tools are essential for maintaining code quality, yet the complexity of writing custom rules often deters developers. We present AutoChecker, an LLM-powered framework that automates the generation of static code checkers using only rule descriptions and test suites. By implementing a Test-Driven Checker Development (TDCD) cycle and a logic-guided API retrieval system, AutoChecker bridges the gap between high-level intent and low-level framework APIs. Our evaluation demonstrates an 82.28% average test pass rate, significantly outperforming standard LLM generation techniques.

Introduction

Static analysis frameworks like PMD and SonarQube are powerful but difficult to customize. Only a small fraction of developers write custom rules due to the steep learning curve associated with massive codebases and complex Abstract Syntax Tree (AST) traversal APIs. While Large Language Models (LLMs) offer potential for automation, they frequently suffer from granularity mismatches and information overload, leading to incomplete logic or API hallucinations.

Methodology

AutoChecker addresses these challenges through two primary innovations:

  1. Test-Driven Checker Development (TDCD): Rather than generating a complete checker in one pass, AutoChecker employs an iterative loop. It processes test cases individually, refining the code to pass each new case while ensuring no regressions occur in previously passed tests.
  2. Logic-guided API-context Retrieval: To prevent API hallucinations, the system utilizes a two-tier database. A "Meta-API" database contains 354 curated "Meta-Operations" (e.g., "Check if method is private"). AutoChecker decomposes high-level rules into these atomic operations before retrieving the exact, verified API snippets required for implementation.
Overview of the AutoChecker workflow, illustrating the iterative development cycle and the integration of framework knowledge.

Evaluation and Results

The framework was evaluated using a benchmark of 20 PMD rules and tested against large-scale open-source projects, including Google Guava and Spring Boot.

  • Performance: AutoChecker achieved an 82.28% average test pass rate with GPT-4, representing a 2.1x to 3.3x improvement over one-shot LLM baselines.
  • Cost-Efficiency: Using models like DeepSeek-V3, a functional checker can be generated for approximately $0.035, a fraction of the cost of manual developer effort.
  • Logic Accuracy: The iterative approach and logic-guided retrieval significantly reduced errors. As shown in the comparison below, standard LLMs often fail to correctly identify the necessary AST nodes or introduce incorrect traversal logic.
Comparison of checker logic generated by standard LLMs (GPT-4) versus AutoChecker against the ground truth PMD implementation.

Discussion and Limitations

The success of AutoChecker highlights the importance of intermediate abstraction layers (Meta-Operations) and iterative refinement in LLM-based software engineering tasks. However, the current implementation is focused on AST-based analysis and is tied to the PMD framework for Java. Future work will explore extending this approach to flow-based analysis and other programming languages and frameworks. Additionally, the system's performance remains dependent on the quality and coverage of the provided test suites.

Summary updated

BibTeX
@misc{xie2024automatically,
  author = {Yuanyuan Xie and Jun Liu and Jiwei Yan and Jinhao Huang and Jun Yan and Jian Zhang},
  title = {{Automatically Write Code Checker: An LLM-based Approach with Logic-guided API Retrieval and Case by Case Iteration}},
  year = {2024},
  eprint = {2411.06796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.06796},
}
Figure