Abstract
In the domain of large-scale software development, the demands for dynamic and multifaceted static code analysis exceed the capabilities of traditional tools. To bridge this gap, we present CodeFuse-Query, a system that redefines static code analysis through the fusion of Domain Optimized System Design and Logic Oriented Computation Design. CodeFuse-Query reimagines code analysis as a data computation task, supporting scanning over 10 billion lines of code daily and more than 300,000 different tasks. It optimizes resource utilization, prioritizes data reusability, applies incremental code extraction, and introduces task types specifically for Code Change, underscoring its domain-optimized design. The system's logic-oriented facet employs Datalog, utilizing a unique two-tiered schema, COREF, to convert source code into data facts. Through Godel, a distinctive language, CodeFuse-Query enables formulation of complex tasks as logical expressions, harnessing Datalog's declarative prowess. This paper provides empirical evidence of CodeFuse-Query's transformative approach, demonstrating its robustness, scalability, and efficiency. We also highlight its real-world impact and diverse applications, emphasizing its potential to reshape the landscape of static code analysis in the context of large-scale software development. Furthermore, in the spirit of collaboration and advancing the field, our project is open-sourced and the repository is available for public access.
Introduction
Static code analysis is critical for maintaining code quality and security in large-scale organizations. However, traditional tools often hit a "complexity and diversity" wall. Systems like Clang Static Analyzer or PMD are often specialized for specific bugs and struggle to scale across thousands of evolving repositories.
Two primary challenges hinder existing state-of-the-art tools like CodeQL:
- Build Dependency: Many tools require a successful project build to perform deep analysis. In massive organizations, "breaking the build" is common, silencing analysis exactly when it is most needed.
- Resource Constraints: Deep path analysis on billions of lines of code is computationally prohibitive without significant optimization.
CodeFuse-Query addresses these by decoupling analysis from the build process and treating the entire codebase as a massive relational database.
System Architecture
CodeFuse-Query is built on a layered architecture designed for massive scalability and high availability. As shown in Figure 1, the system consists of four primary layers:
- Access Layer: Manages user requests via proxies and caches.
- Coordinator Layer: Handles task scheduling and cluster management through the Analysis Coordinator and DataSentinel.
- Worker Layer: Executes analysis and extraction tasks using dedicated nodes and the CoQuery-Runtime.
- Storage Layer: Persists code facts in RDBMS and Object Storage.
This domain-optimized design allows the system to handle over 300,000 tasks daily with a 100% analysis success rate by utilizing build-independent extraction.
Logic-Oriented Computation Design
The core of CodeFuse-Query's analytical power lies in its logic-oriented approach, which transforms source code into queryable data facts.
The COREF Data Model
Instead of a standard Abstract Syntax Tree (AST), the system uses a two-tiered schema called COREF:
- Tier 1 (EDB): Stores raw, compact facts extracted directly from source files (Extensional Database).
- Tier 2 (IDB): Consists of derived facts calculated at execution time (Intensional Database), representing higher-level relationships.
The GÖDEL Language
Analysis tasks are written in GÖDEL, a custom Domain-Specific Language (DSL) built on Datalog. GÖDEL provides a declarative interface for simple queries while supporting complex algorithms through Rust-like constructs. Its Turing-complete nature and support for recursion are vital for analyzing deep code dependencies.
As illustrated in Figure 2, the computation process involves interpreting user queries, extracting facts from source code into the COREF model, and generating an optimized execution plan that is compiled into an analysis executable.
Evaluation and Key Results
The system was evaluated using 50 diverse open-source repositories and internal data from Ant Group, where it supports over 10,000 developers.
- Efficiency and Speed: For Java, extraction was 5.66x faster than CodeQL.
- Build Independence: CodeFuse-Query achieved a 100% success rate in analyzing Java repositories, whereas CodeQL failed on nearly 72% due to build environment issues.
- Incremental Analysis: The system introduces Delta Code Analysis (DCA), which requires only 3.67% of the time of a full repository scan. Incremental Full Repository Analysis (IFRA) showed an average 82.2% reduction in extraction time and an 84.4% reduction in memory usage.
- Data Reusability: Cached COREF facts have a reuse factor of approximately 13x, meaning the expensive work of code extraction is amortized across multiple security and compliance tasks.
Conclusion
CodeFuse-Query demonstrates that treating static analysis as a data computation problem enables unprecedented scale and reliability. By removing build dependencies and utilizing a logic-oriented schema, the system successfully scans 10 billion lines of code daily. Future work includes leveraging CodeFuse-Query to prepare high-quality datasets for training specialized Code-LLMs and improving the reusability of specific query results.
Summary updated