ProConSuL: Project Context for Code Summarization with LLMs

Vadim Lomshakov, Andrey Podivilov, Sergey Savin, Oleg Baryshnikov, Alena Lisevych, Sergey I. NikolenkoEMNLP 2024 · DOI

Introduction

Understanding a single function in a complex software project often requires knowledge of its dependencies—the functions it calls (callees). While human developers naturally navigate call graphs to build mental models, standard Large Language Models (LLMs) typically process functions in isolation. This limitation is particularly acute in C/C++, a language critical to global infrastructure but notoriously difficult for automated analysis due to its complexity and the prevalence of low-quality, "noisy" docstrings in existing datasets.

PROCONSUL (Project Context for Code Summarization) addresses these challenges by providing LLMs with precise project context. By focusing on recursive summarization and specialized training, the framework enables smaller, open-source models to achieve performance levels comparable to state-of-the-art closed models like GPT-4o.

The PROCONSUL Framework

The PROCONSUL framework employs a multi-step pipeline to generate high-quality code summaries:

  1. Formal Code Analysis: Utilizing Clang-based tools, the framework constructs a global call graph for the target project.
  2. Recursive Summarization: The system summarizes dependencies starting from the leaf nodes of the call graph. These summaries are then injected into the prompt as context when summarizing "parent" functions.
  3. Two-Phase Training:
    • Supervised Fine-Tuning (SFT): Models are trained on high-quality synthetic data generated by GPT-4o to establish a consistent summary style and teach the model to utilize the provided context.
  4. Preference Alignment (ORPO): Using Odds Ratio Preference Optimization, the model is trained to prefer concise, informative summaries over verbose or "trivial" ones (summaries that merely repeat the function signature).
  5. Context Obfuscation: To ensure the model actually "reads" the provided context rather than relying on descriptive function names, the training process includes a phase where function names are replaced with random strings.
Figure 2: Comparison of summaries generated by different PROCONSUL variants and the base CodeLlama model, illustrating the impact of context and obfuscation.

Evaluation and Key Results

The authors evaluated PROCONSUL using a new benchmark of five popular C/C++ repositories (including openssl and ffmpeg) not seen during training. The evaluation focused on several criteria: sufficiency (whether the summary is enough to understand the function), factual correctness, and the presence of hallucinations or trivial information.

Performance Gains

PROCONSUL-7B demonstrated significant improvements over the base CodeLlama-7B model. In head-to-head comparisons, PROCONSUL's summaries were preferred in 20 out of 25 cases. Most notably, the model reduced hallucinations to zero in the manual test set, a critical requirement for reliable software documentation.

Figure 6: Quantitative results comparing PROCONSUL variants against baselines across multiple metrics, including sufficiency, correctness, and hallucinations.

Scaling and Context Value

Manual analysis revealed that "Callees" are the most valuable context type, critical in 46% of cases. The framework also scaled effectively to the 34B-parameter version of CodeLlama, which yielded even higher factual correctness. A key insight from the study was that training on high-quality synthetic data proved superior to using original human-written docstrings, which are often inconsistent or missing.

Conclusion

PROCONSUL demonstrates that providing precise, recursively-generated project context allows 7B-parameter models to rival much larger models in code summarization tasks. By focusing on C/C++ and addressing the "noisy reference" problem, this work provides a blueprint for building LLMs capable of comprehending entire, multi-million-line software repositories. Future work will focus on integrating additional context types, such as global variables and class definitions, more deeply into the automated pipeline.

Summary updated

BibTeX
@inproceedings{lomshakov2024proconsul,
  author = {Vadim Lomshakov and Andrey Podivilov and Sergey Savin and Oleg Baryshnikov and Alena Lisevych and Sergey I. Nikolenko},
  title = {{ProConSuL: Project Context for Code Summarization with LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-industry.65},
}
Figure