Code-Survey: An LLM-Driven Methodology for Analyzing Large-Scale Codebases

Yusheng Zheng, Yiwei Yang, Haoqin Tu, Yuxi HuangarXiv 2024 · DOI · arXiv

Introduction

Modern software systems, such as the Linux kernel, have reached a scale that exceeds the cognitive capacity of any individual human expert. While static and dynamic analysis tools excel at identifying code patterns, they often fail to capture the underlying design rationale—the "why" behind specific changes. This critical "tribal knowledge" is typically buried in unstructured text, including thousands of commit logs, mailing list discussions, and developer email threads. Manually synthesizing this data to understand long-term evolution or identify systemic risks is a Herculean task.

To address this, we present Code-Survey, a novel methodology that treats Large Language Models (LLMs) as participants in a sociological survey. By transforming unstructured developer communications into structured, analyzable datasets, Code-Survey enables researchers to quantitatively track software evolution and uncover hidden design trends.

Methodology

The core of Code-Survey is a workflow that mimics traditional social science survey techniques, adapted for the scale of modern codebases.

The Code-Survey workflow involves iterative design, LLM-driven survey completion, and quantitative analysis.

The process consists of four primary stages:

  1. Survey Design: Human experts or LLM agents define targeted, structured questions for the data (e.g., "Is this a bug fix?", "What is the complexity of this change?").
  2. Survey Completion: LLM agents (e.g., GPT-4o) act as respondents, processing commit messages and code diffs to provide structured JSON answers.
  3. Evaluation & Refinement: Results are sampled and validated by humans to mitigate hallucinations and ensure consistency.
  4. Quantitative Analysis: The resulting structured data is analyzed using statistical methods to identify patterns and hotspots.

The Linux-bpf Dataset

To demonstrate the methodology, we introduced the Linux-bpf dataset, focusing on the eBPF (extended Berkeley Packet Filter) subsystem. This dataset encompasses over 670 features, 15,000+ commits spanning eight years, and 150,000+ related emails. Unlike traditional datasets, it includes LLM-generated categorizations for every commit, providing a granular view of the subsystem's lifecycle.

Key Insights and Results

The application of Code-Survey to the Linux kernel yielded several significant insights:

The Shift to kfuncs

One of the most striking trends identified was the evolution of kernel integration methods. While traditional "helper functions" have reached a plateau, there has been an explosion in the adoption of "kfuncs" (kernel functions) since 2021. This shift indicates a move toward more flexible and deeper kernel integrations within the eBPF community.

Cumulative count of Helper vs. Kfunk features over time, showing the recent surge in kfuncs.

Bug Hotspots and Maintenance

Contrary to the heavy academic focus on the eBPF verifier's safety, Code-Survey revealed that approximately 35% of bugs originate from event-related code and the syscall interface. The study identified kernel/bpf/verifier.c as the most bug-prone file (425 bugs), followed by net/core/filter.c (140 bugs). Furthermore, analysis of commit types showed a transition from a "feature-heavy" growth phase (2020-2021) to a "stability and cleanup" phase starting in 2022.

Reliability and Validation

The methodology demonstrated high reliability, with a misclassification rate of under 0.05% for noise filtering. Five eBPF maintainers validated the findings, confirming that the AI-generated insights accurately reflected their real-world experience of the subsystem’s evolution.

Limitations and Future Work

The effectiveness of Code-Survey is inherently tied to the quality of the underlying documentation; cryptic commit messages limit the LLM's context. Additionally, while execution is automated, the initial survey design still requires domain expertise. Future work will involve expanding the methodology to other large-scale projects like Kubernetes and LLVM, and integrating richer data sources such as execution traces and code flow graphs.

Summary updated

BibTeX
@misc{zheng2024code,
  author = {Yusheng Zheng and Yiwei Yang and Haoqin Tu and Yuxi Huang},
  title = {{Code-Survey: An LLM-Driven Methodology for Analyzing Large-Scale Codebases}},
  year = {2024},
  eprint = {2410.01837},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.01837},
}
Figure