Introduction
Modern software systems, such as the Linux kernel, have reached a scale that exceeds the cognitive capacity of any individual human expert. While static and dynamic analysis tools excel at identifying code patterns, they often fail to capture the underlying design rationale—the "why" behind specific changes. This critical "tribal knowledge" is typically buried in unstructured text, including thousands of commit logs, mailing list discussions, and developer email threads. Manually synthesizing this data to understand long-term evolution or identify systemic risks is a Herculean task.
To address this, we present Code-Survey, a novel methodology that treats Large Language Models (LLMs) as participants in a sociological survey. By transforming unstructured developer communications into structured, analyzable datasets, Code-Survey enables researchers to quantitatively track software evolution and uncover hidden design trends.
Methodology
The core of Code-Survey is a workflow that mimics traditional social science survey techniques, adapted for the scale of modern codebases.
The process consists of four primary stages:
- Survey Design: Human experts or LLM agents define targeted, structured questions for the data (e.g., "Is this a bug fix?", "What is the complexity of this change?").
- Survey Completion: LLM agents (e.g., GPT-4o) act as respondents, processing commit messages and code diffs to provide structured JSON answers.
- Evaluation & Refinement: Results are sampled and validated by humans to mitigate hallucinations and ensure consistency.
- Quantitative Analysis: The resulting structured data is analyzed using statistical methods to identify patterns and hotspots.
The Linux-bpf Dataset
To demonstrate the methodology, we introduced the Linux-bpf dataset, focusing on the eBPF (extended Berkeley Packet Filter) subsystem. This dataset encompasses over 670 features, 15,000+ commits spanning eight years, and 150,000+ related emails. Unlike traditional datasets, it includes LLM-generated categorizations for every commit, providing a granular view of the subsystem's lifecycle.
Key Insights and Results
The application of Code-Survey to the Linux kernel yielded several significant insights:
The Shift to kfuncs
One of the most striking trends identified was the evolution of kernel integration methods. While traditional "helper functions" have reached a plateau, there has been an explosion in the adoption of "kfuncs" (kernel functions) since 2021. This shift indicates a move toward more flexible and deeper kernel integrations within the eBPF community.
Bug Hotspots and Maintenance
Contrary to the heavy academic focus on the eBPF verifier's safety, Code-Survey revealed that approximately 35% of bugs originate from event-related code and the syscall interface. The study identified kernel/bpf/verifier.c as the most bug-prone file (425 bugs), followed by net/core/filter.c (140 bugs). Furthermore, analysis of commit types showed a transition from a "feature-heavy" growth phase (2020-2021) to a "stability and cleanup" phase starting in 2022.
Reliability and Validation
The methodology demonstrated high reliability, with a misclassification rate of under 0.05% for noise filtering. Five eBPF maintainers validated the findings, confirming that the AI-generated insights accurately reflected their real-world experience of the subsystem’s evolution.
Limitations and Future Work
The effectiveness of Code-Survey is inherently tied to the quality of the underlying documentation; cryptic commit messages limit the LLM's context. Additionally, while execution is automated, the initial survey design still requires domain expertise. Future work will involve expanding the methodology to other large-scale projects like Kubernetes and LLVM, and integrating richer data sources such as execution traces and code flow graphs.
Summary updated