Introduction
Software development is inherently a collaborative effort that relies on shared mental models. These models are frequently communicated through informal visual artifacts, such as whiteboard sketches or architectural diagrams. However, maintaining these visuals is a significant challenge; as source code evolves, documentation often becomes "stale," leading to a disconnect between the system's design and its implementation. While Large Language Models (LLMs) have demonstrated remarkable proficiency in generating textual documentation, they often struggle to produce structured, high-level visual representations directly from complex codebases. Furthermore, evaluating the quality of such diagrams is traditionally subjective and requires human-authored ground truth, which is rarely available.
To address these challenges, this paper introduces VisDocSketcher, a multi-agent framework designed to automate the generation of visual documentation, and AutoSketchEval, an evaluation framework that assesses diagram quality through the lens of information reconstruction.
The VisDocSketcher Architecture
VisDocSketcher moves away from the "single-prompt" paradigm, instead employing a specialized multi-agent system to handle the nuances of code visualization. This modular approach allows for a more robust transformation from raw source code to renderable diagrams.
The system consists of several key roles:
- Supervisor: Acts as the orchestrator, managing the overall workflow and filesystem interactions.
- Analyser: Conducts static analysis on the source code to identify critical data flows and variable dependencies.
- Sketcher: Translates the extracted analysis into Mermaid.js, a text-based diagramming language that can be easily rendered into visuals.
- Repair Agent: Monitors the output of the Sketcher to identify and correct syntax errors, ensuring the final Mermaid code is valid.
- Visuals Agent: Enhances the diagram's readability by adding human-centric elements such as color-coding, icons, and emojis to distinguish between different components like data sources and models.
AutoSketchEval: Evaluation via Reconstruction
A core contribution of this work is AutoSketchEval, a framework that treats visual documentation as a form of lossy compression. By measuring how much information is preserved in a diagram, the system can evaluate quality without needing a human reference.
The evaluation process follows a three-step cycle:
- Encoding (Code2Sketch): An agent generates a diagram from the original source code.
- Decoding (Sketch2Code): A separate agent attempts to reconstruct the original code based solely on the information provided in the diagram.
- Comparison: The original and reconstructed code are compared using metrics such as CodeBLEU and CodeBERTScore.
High reconstructability serves as a proxy for high-quality documentation, indicating that the sketch successfully captured the essential logic of the code.
Experimental Results and Insights
The researchers evaluated VisDocSketcher using two primary datasets: a high-quality set of 19 Jupyter notebooks with human-authored diagrams and DistillKaggle, a collection of 1,000 notebooks representing diverse levels of developer expertise.
Key findings include:
- Performance: VisDocSketcher successfully generated renderable documentation for 74.4% of the notebooks. The multi-agent approach outperformed standard template-based baselines by 26.7% to 39.8%.
- The Reconstruction Proxy: The AutoSketchEval method proved highly effective, achieving an AUC of 0.87 in distinguishing between high- and low-quality documentation.
- Complexity and Expertise: The study revealed a "complexity tax," where dataflow accuracy in sketches dropped by approximately 8% for every additional 100 lines of code. Interestingly, code written by "Grandmaster" developers was harder to visualize than beginner code, likely due to more abstract logic.
- Efficiency Trade-offs: While the multi-agent system produced superior results, it was significantly slower—taking roughly 93 seconds per notebook compared to 8 seconds for a single-agent approach.
Conclusion
VisDocSketcher demonstrates that agentic systems can bridge the gap between source code and high-level visual documentation. By leveraging the AutoSketchEval framework, developers can automatically validate the utility of these diagrams. Future work will focus on improving the aesthetic quality of the generated sketches and exploring their impact on developer onboarding in large-scale software architectures.
Summary updated