Context Matters: Improving LLM-Based Code Summarization with Hierarchical Information

Alexandre Petrachini, Breno Bernard Nicolau de França, Guido AraujoarXiv 2026 · arXiv

TL;DR

Providing Large Language Models (LLMs) with hierarchical context—such as the enclosing class or module—significantly improves the quality of automated code summaries. By keeping surrounding documentation intact rather than stripping it, developers can generate summaries that more closely align with human intent, with Google's Gemini model showing the most substantial gains from this extra information.

What Problem Does This Paper Address?

Software documentation is a double-edged sword: it is essential for maintenance and bug fixing, yet it is frequently missing, outdated, or focused too narrowly on local implementation details. While LLMs have become the go-to tool for "code summarization" (turning a function's logic into natural language), most current tools only look at the function body itself.

This "local-only" view ignores the broader context—like shared class attributes or the purpose of the overall module—that a human developer naturally uses to understand code. The authors set out to determine if "zooming out" to include hierarchical structure and existing neighboring documentation actually helps LLMs produce better summaries, or if it just wastes the model's limited context window.

Methods

The researchers conducted an in silico study using an incomplete factorial design to test two main variables:

  1. Context Granularity: How much code does the model see? (Function-only, Class-level, or Module-level).
  2. Documentation Mode: Does the model see existing docstrings in that context?
  3. Documentation-Aware (DA): Keeps docstrings for neighboring methods and classes.
  4. Documentation-Independent (DI): Strips all existing documentation, leaving only raw code.

They tested three popular LLMs: GPT-4o-mini, Gemini-1.5-Flash (referred to as Gemini-3-Flash in the text), and DeepSeek v3.2. To evaluate the results, they used standard metrics (BLEU, ROUGE, METEOR), semantic similarity (BERTScore), and a "LLM-as-a-Judge" approach where Claude 3.5 Sonnet ranked the summaries based on adequacy, conciseness, and fluency.

Datasets

The study utilized 12 mature Python open-source repositories sourced from the SWE-bench dataset, including well-known projects like django, flask, matplotlib, and scikit-learn.

To ensure a rigorous comparison, they filtered the data to include only methods that were already documented at all three levels (function, class, and module), resulting in a final dataset of 6,932 methods.

Key Insights and Ideas

  • The "Gemini" Advantage: Not all models are created equal when it comes to context. Gemini showed a "medium" effect size improvement when given hierarchical context, whereas GPT-4o-mini and DeepSeek showed "weak" improvements. This suggests Gemini is particularly adept at utilizing surrounding structural cues.
  • Don't Strip the Docstrings: A major finding was that Documentation-Aware (DA) prompts consistently outperformed Documentation-Independent (DI) ones. Keeping neighboring documentation helps the model disambiguate the intent of the code, rather than just describing the syntax.
  • Class-Level is the Sweet Spot: For most models, performance peaked at the Class-level DA context. Adding the entire module sometimes added noise or "wasted" context without providing a proportional boost in summary quality.
  • Style vs. Substance: The authors noted that Gemini often adopted the documentation style of the surrounding code. This "stylistic alignment" helped it score higher on similarity metrics compared to the more verbose DeepSeek, which tended to produce summaries six times longer than the human reference.

Key Results

  • Statistically Significant Gains: Adding hierarchical context and surrounding documentation yielded improvements across all models and all four metrics (p < .001).
  • Judge Preference: The Claude-based judge preferred Documentation-Aware (DA) summaries over Documentation-Independent (DI) ones in over 50% of cases across all models.
  • Effect Sizes: Gemini showed the strongest response to context (r = 0.31–0.36), while GPT-4o-mini (r = 0.07–0.26) and DeepSeek (r = 0.04–0.18) were more modest.
  • Human vs. AI: Interestingly, while LLM summaries are generally much longer than human ones, the judge tended to prefer human references more as they got longer and more detailed, suggesting that "completeness" is the primary gap LLMs need to bridge.

Limitations and Future Work

  • Language Specificity: The study was limited to Python. Different languages with different documentation conventions (like Javadoc or Doxygen) might yield different results.
  • Scope of Context: The study focused on immediate hierarchical context (parents/siblings). It did not explore call-graph traversal (following a function's dependencies into other files) or repository-wide retrieval-augmented generation (RAG).
  • Data Contamination: As the repositories are public, the models likely saw this code during training. While the authors argue the models aren't just "recalling" (due to the length differences in output), testing on code committed after the models' knowledge cutoffs remains a necessary next step.
  • Future Directions: The authors suggest repeating the study with smaller open-weight models (7B/13B) to see if context sensitivity scales with model size.

Summary updated

Abstract

Source code documentation plays an important role in software quality activities, such as code review, program comprehension, maintenance, and bug fixing. However, it is often fragmented across individual code elements and focuses on local implementation details, providing little information about their surrounding context. This limits both developers' understanding of software systems and the effectiveness of Large Language Models (LLMs), whose reasoning is constrained by finite context windows. Enriching documentation with hierarchical context has the potential to improve software comprehension and support downstream, quality-related tasks for both developers and LLM-based tools. In this paper, we present an in silico study that showcases the impact of augmenting context on the generation of code-level documentation, using a variety of LLMs. By varying the function, class, and module hierarchy visible to the LLMs, we assess whether this variation produces a significant difference in the resulting code documentation compared with the existing documentation, which serves as a proxy for developer intent. Our results show that Documentation-Aware prompts with class or module-level context consistently yield summaries that are closest to the original documentation. Adding surrounding documentation yields statistically significant improvements over the baseline across models, with effect sizes ranging from negligible (DeepSeek) through weak (GPT-4o-mini) to medium (Gemini). We restrict the study to this immediate hierarchical context; call-graph traversal and cross-module dependency context fall outside our scope and are left to future work. These findings provide a clear path to improve automated documentation tools and enhance LLMs' ability to reason about codebases for downstream tasks, such as code retrieval.
BibTeX
@misc{petrachini2026context,
  author = {Alexandre Petrachini and Breno Bernard Nicolau de França and Guido Araujo},
  title = {{Context Matters: Improving LLM-Based Code Summarization with Hierarchical Information}},
  year = {2026},
  eprint = {2611.18964},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18964},
}
Figure