TL;DR
Providing Large Language Models (LLMs) with hierarchical context—such as the enclosing class or module—significantly improves the quality of automated code summaries. By keeping surrounding documentation intact rather than stripping it, developers can generate summaries that more closely align with human intent, with Google's Gemini model showing the most substantial gains from this extra information.
What Problem Does This Paper Address?
Software documentation is a double-edged sword: it is essential for maintenance and bug fixing, yet it is frequently missing, outdated, or focused too narrowly on local implementation details. While LLMs have become the go-to tool for "code summarization" (turning a function's logic into natural language), most current tools only look at the function body itself.
This "local-only" view ignores the broader context—like shared class attributes or the purpose of the overall module—that a human developer naturally uses to understand code. The authors set out to determine if "zooming out" to include hierarchical structure and existing neighboring documentation actually helps LLMs produce better summaries, or if it just wastes the model's limited context window.
Methods
The researchers conducted an in silico study using an incomplete factorial design to test two main variables:
- Context Granularity: How much code does the model see? (Function-only, Class-level, or Module-level).
- Documentation Mode: Does the model see existing docstrings in that context?
- Documentation-Aware (DA): Keeps docstrings for neighboring methods and classes.
- Documentation-Independent (DI): Strips all existing documentation, leaving only raw code.
They tested three popular LLMs: GPT-4o-mini, Gemini-1.5-Flash (referred to as Gemini-3-Flash in the text), and DeepSeek v3.2. To evaluate the results, they used standard metrics (BLEU, ROUGE, METEOR), semantic similarity (BERTScore), and a "LLM-as-a-Judge" approach where Claude 3.5 Sonnet ranked the summaries based on adequacy, conciseness, and fluency.
Datasets
The study utilized 12 mature Python open-source repositories sourced from the SWE-bench dataset, including well-known projects like django, flask, matplotlib, and scikit-learn.
To ensure a rigorous comparison, they filtered the data to include only methods that were already documented at all three levels (function, class, and module), resulting in a final dataset of 6,932 methods.
Key Insights and Ideas
- The "Gemini" Advantage: Not all models are created equal when it comes to context. Gemini showed a "medium" effect size improvement when given hierarchical context, whereas GPT-4o-mini and DeepSeek showed "weak" improvements. This suggests Gemini is particularly adept at utilizing surrounding structural cues.
- Don't Strip the Docstrings: A major finding was that Documentation-Aware (DA) prompts consistently outperformed Documentation-Independent (DI) ones. Keeping neighboring documentation helps the model disambiguate the intent of the code, rather than just describing the syntax.
- Class-Level is the Sweet Spot: For most models, performance peaked at the Class-level DA context. Adding the entire module sometimes added noise or "wasted" context without providing a proportional boost in summary quality.
- Style vs. Substance: The authors noted that Gemini often adopted the documentation style of the surrounding code. This "stylistic alignment" helped it score higher on similarity metrics compared to the more verbose DeepSeek, which tended to produce summaries six times longer than the human reference.
Key Results
- Statistically Significant Gains: Adding hierarchical context and surrounding documentation yielded improvements across all models and all four metrics (p < .001).
- Judge Preference: The Claude-based judge preferred Documentation-Aware (DA) summaries over Documentation-Independent (DI) ones in over 50% of cases across all models.
- Effect Sizes: Gemini showed the strongest response to context (r = 0.31–0.36), while GPT-4o-mini (r = 0.07–0.26) and DeepSeek (r = 0.04–0.18) were more modest.
- Human vs. AI: Interestingly, while LLM summaries are generally much longer than human ones, the judge tended to prefer human references more as they got longer and more detailed, suggesting that "completeness" is the primary gap LLMs need to bridge.
Limitations and Future Work
- Language Specificity: The study was limited to Python. Different languages with different documentation conventions (like Javadoc or Doxygen) might yield different results.
- Scope of Context: The study focused on immediate hierarchical context (parents/siblings). It did not explore call-graph traversal (following a function's dependencies into other files) or repository-wide retrieval-augmented generation (RAG).
- Data Contamination: As the repositories are public, the models likely saw this code during training. While the authors argue the models aren't just "recalling" (due to the length differences in output), testing on code committed after the models' knowledge cutoffs remains a necessary next step.
- Future Directions: The authors suggest repeating the study with smaller open-weight models (7B/13B) to see if context sensitivity scales with model size.
Summary updated