Introduction
Understanding a single function in a complex software project often requires knowledge of its dependencies—the functions it calls (callees). While human developers naturally navigate call graphs to build mental models, standard Large Language Models (LLMs) typically process functions in isolation. This limitation is particularly acute in C/C++, a language critical to global infrastructure but notoriously difficult for automated analysis due to its complexity and the prevalence of low-quality, "noisy" docstrings in existing datasets.
PROCONSUL (Project Context for Code Summarization) addresses these challenges by providing LLMs with precise project context. By focusing on recursive summarization and specialized training, the framework enables smaller, open-source models to achieve performance levels comparable to state-of-the-art closed models like GPT-4o.
The PROCONSUL Framework
The PROCONSUL framework employs a multi-step pipeline to generate high-quality code summaries:
- Formal Code Analysis: Utilizing Clang-based tools, the framework constructs a global call graph for the target project.
- Recursive Summarization: The system summarizes dependencies starting from the leaf nodes of the call graph. These summaries are then injected into the prompt as context when summarizing "parent" functions.
- Two-Phase Training:
- Supervised Fine-Tuning (SFT): Models are trained on high-quality synthetic data generated by GPT-4o to establish a consistent summary style and teach the model to utilize the provided context.
- Preference Alignment (ORPO): Using Odds Ratio Preference Optimization, the model is trained to prefer concise, informative summaries over verbose or "trivial" ones (summaries that merely repeat the function signature).
- Context Obfuscation: To ensure the model actually "reads" the provided context rather than relying on descriptive function names, the training process includes a phase where function names are replaced with random strings.
Evaluation and Key Results
The authors evaluated PROCONSUL using a new benchmark of five popular C/C++ repositories (including openssl and ffmpeg) not seen during training. The evaluation focused on several criteria: sufficiency (whether the summary is enough to understand the function), factual correctness, and the presence of hallucinations or trivial information.
Performance Gains
PROCONSUL-7B demonstrated significant improvements over the base CodeLlama-7B model. In head-to-head comparisons, PROCONSUL's summaries were preferred in 20 out of 25 cases. Most notably, the model reduced hallucinations to zero in the manual test set, a critical requirement for reliable software documentation.
Scaling and Context Value
Manual analysis revealed that "Callees" are the most valuable context type, critical in 46% of cases. The framework also scaled effectively to the 34B-parameter version of CodeLlama, which yielded even higher factual correctness. A key insight from the study was that training on high-quality synthetic data proved superior to using original human-written docstrings, which are often inconsistent or missing.
Conclusion
PROCONSUL demonstrates that providing precise, recursively-generated project context allows 7B-parameter models to rival much larger models in code summarization tasks. By focusing on C/C++ and addressing the "noisy reference" problem, this work provides a blueprint for building LLMs capable of comprehending entire, multi-million-line software repositories. Future work will focus on integrating additional context types, such as global variables and class definitions, more deeply into the automated pipeline.
Summary updated