Chain of Understanding: Supporting Code Understanding with Large Language Models

Jie Gao, Yue Xue, Xiaofei Xie, SoeMin Thant, Erika LeearXiv 2025 · arXiv · OpenReview

Abstract

Code auditing demands a robust understanding of codebases - an especially challenging task for end-user developers with limited expertise. To address this, we conducted formative interviews with experienced auditors and identified a Chain-of-Understanding approach, in which Large Language Models (LLMs) guide developers through hierarchical code comprehension - from high-level overviews to specific functions and variables. Building on this, we incorporated the Chain-of-Understanding concept into CodeMap, a system offering interactive visualizations, stepwise guided analysis, and context-aware chatbot support. Through within-subject user studies with 10 participants of diverse backgrounds and 5 expert and 2 novice interviews, CodeMap proved effective in reducing the manual effort of prompt engineering while enhancing engagement with visualization, outperforming both standalone LLMs and traditional static visualization tools.

Introduction

Software developers spend approximately 58% of their time attempting to understand existing codebases. While Large Language Models (LLMs) have revolutionized snippet-level explanation, they often struggle to convey the "big picture" of a complex project. Current AI-assisted tools are primarily chatbot-centric, forcing users into linear, text-heavy interactions that do not align with the non-linear, exploratory nature of human code comprehension. This research bridges that gap by studying professional code auditors—experts who rapidly master unfamiliar codebases—to automate their mental models into a "Chain of Understanding."

Methodology

The development of CodeMap followed a rigorous four-stage Human-Computer Interaction (HCI) workflow, moving from expert insights to a validated prototype.

The four-stage research methodology used to develop and validate CodeMap.
  1. Formative Interviews: Eight professional auditors were interviewed to extract their "tacit knowledge." They identified a top-down flow: Global overview $\rightarrow$ Local module relationships $\rightarrow$ Detailed logic.
  2. Design Opportunities: The team identified four critical needs: automated information extraction, project decomposition, multi-layer representation, and reduced prompt engineering effort.
  3. Prototype Development (CodeMap): A web-based tool was built using a Retrieval-Augmented Generation (RAG) backend powered by GPT-4o. CodeMap generates Global Business Maps, Local Maps, and Architectural Reading Guides.
  4. Iterative Refinement: To combat LLM hallucinations, a feedback loop was implemented where the LLM reviews and corrects its own generated maps across multiple iterations.

The CodeMap System and Key Insights

CodeMap codifies the "Auditor" mental model, which prioritizes understanding the business problem and data flow before examining specific function definitions. By providing structured visuals, CodeMap allows developers to offload cognitive burden. Instead of mentally tracking file connections, users can interact with a visual representation, freeing mental resources for logic analysis.

Human-LLM co-understanding framework mapping human activities to LLM system support.

This "scaffolding" is particularly vital for novices, preventing "unpurposeful exploration"—the tendency to get lost in irrelevant utility files. The system bridges the gap between human activities (like constructing mental models) and LLM support (like providing structural representations).

Evaluation and Results

The system was evaluated against 20 open-source projects (Python, Solidity, Java) and through a user study with 15 participants (9 experienced, 6 novices).

  • Reduced Text Reliance: Experienced developers spent 79% less time reading and interpreting LLM text responses compared to using a standard chatbot.
  • Increased Engagement: Users spent 90% more time interacting with the system-provided maps than they did with traditional static visualization tools.
  • High Accuracy: Through iterative refinement, CodeMap achieved 99% accuracy in mapping files for medium-sized Python projects.
  • Subjective Preference: 100% of experienced developers found CodeMap more intuitive than ChatGPT or traditional static analysis tools for codebase-level understanding.

Conclusion

CodeMap demonstrates that a hierarchical, visual-first approach significantly improves codebase comprehension. While context window limitations remain a challenge for massive repositories (>100k lines), future advancements in LLM capacity and deeper IDE integration will likely further enhance these tools. This research highlights that the key to effective AI assistance lies in mimicking expert mental models rather than just providing conversational interfaces.

Summary updated

Abstract

Code auditing demands a robust understanding of codebases - an especially challenging task for end-user developers with limited expertise. To address this, we conducted formative interviews with experienced auditors and identified a Chain-of-Understanding approach, in which Large Language Models (LLMs) guide developers through hierarchical code comprehension - from high-level overviews to specific functions and variables. Building on this, we incorporated the Chain-of-Understanding concept into CodeMap, a system offering interactive visualizations, stepwise guided analysis, and context-aware chatbot support. Through within-subject user studies with 10 participants of diverse backgrounds and 5 expert and 2 novice interviews, CodeMap proved effective in reducing the manual effort of prompt engineering while enhancing engagement with visualization, outperforming both standalone LLMs and traditional static visualization tools.
BibTeX
@misc{gao2025chain,
  author = {Jie Gao and Yue Xue and Xiaofei Xie and SoeMin Thant and Erika Lee},
  title = {{Chain of Understanding: Supporting Code Understanding with Large Language Models}},
  year = {2025},
  eprint = {2504.04553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.04553},
}
Figure