XGraphRAG: Interactive Visual Analysis for Graph-based Retrieval-Augmented Generation

Ke Wang, Bo Pan, Yingchaojie Feng, Yuwei Wu, Jieyi Chen, Minfeng Zhu, Wei Chen

Abstract

Graph-based Retrieval-Augmented Generation (RAG) has shown great capability in enhancing Large Language Model (LLM)'s answer with an external knowledge base. Compared to traditional RAG, it introduces a graph as an intermediate representation to capture better structured relational knowledge in the corpus, elevating the precision and comprehensiveness of generation results. However, developers usually face challenges in analyzing the effectiveness of GraphRAG on their dataset due to GraphRAG's complex information processing pipeline and the overwhelming amount of LLM invocations involved during graph construction and query, which limits GraphRAG interpretability and accessibility. This research proposes a visual analysis framework that helps RAG developers identify critical recalls of GraphRAG and trace these recalls through the GraphRAG pipeline. Based on this framework, we develop XGraphRAG, a prototype system incorporating a set of interactive visualizations to facilitate users' analysis process, boosting failure cases collection and improvement opportunities identification. Our evaluation demonstrates the effectiveness and usability of our approach.

Introduction

Retrieval-Augmented Generation (RAG) has become the standard for grounding Large Language Models (LLMs) in private or domain-specific data. Recently, GraphRAG has emerged as a powerful evolution of this technique. By using knowledge graphs to capture complex relationships that standard vector-based RAG often misses, GraphRAG provides more structured and comprehensive answers.

However, this power comes at the cost of transparency. GraphRAG is a "black box" on steroids. Its pipeline is significantly more complex than traditional RAG, involving document slicing, entity and triplet extraction, node merging, hierarchical topic partitioning, and multi-stage retrieval. When a GraphRAG system produces an error, developers are often "lost in a sea of LLM calls," unable to determine if the failure occurred during the initial graph construction, the retrieval phase, or the final reasoning step. This research introduces XGraphRAG, a visual analysis framework designed to bridge this gap in traceability and interpretability.

The XGraphRAG Framework

XGraphRAG implements a two-stage visual analysis workflow to help developers debug and optimize their systems:

1. Identifying Suspicious Retrievals

The system utilizes an LLM-assisted framework to reconstruct the "inference path" of both the generated answer and a "ground truth" answer. By comparing these paths side-by-side, XGraphRAG flags Suspicious Recalls. These are categorized into:

  • Missing Recalls: Facts present in the knowledge base that the LLM should have used but failed to retrieve.
  • Unexpected Recalls: Irrelevant or incorrect facts that led the LLM astray during the generation process.

2. Root Cause Analysis

Once a suspicious recall is identified, users can utilize four specialized interactive views to perform a deep dive into the system's logic:

  • QA & Inference Trace View: Provides a side-by-side comparison of reasoning steps, allowing developers to see the "evidence facts" the LLM claimed to use.
  • Topic Explore View: Employs circle packing visualizations to represent the global hierarchy of the graph. This helps users determine if errors stem from how topics were summarized or partitioned.
  • Entity Explore View: A node-link diagram for local relationship analysis, showing how specific entities are connected and identifying potential missing links.
  • LLM Invocation View: A granular "drill-down" tool that displays the exact prompt and output for any specific step in the pipeline, from extraction to summarization.

Key Insights

  • The Power of the Inference Trace: By reconstructing the reasoning chain, developers can verify the LLM's claims against the actual graph data. This makes it possible to trace an error in the final answer back to the specific raw text chunk from which a fact was extracted.
  • Multiscale Analysis: GraphRAG's strength lies in its hierarchy. XGraphRAG allows users to jump between "global" topic summaries and "local" entity details, revealing cases where a high-level summary might have omitted a crucial low-level detail.

Evaluation and Results

The system was evaluated using the MultiHop-RAG news dataset, which tests an agent's ability to connect information across multiple documents. A user study was conducted with 14 RAG engineers, comparing XGraphRAG against Kotaemon, a popular open-source GraphRAG interface.

  • Accuracy: XGraphRAG significantly outperformed the baseline. For complex tasks, such as identifying faulty topic reports or missing relationships, users had a 0% success rate with the baseline but achieved over 85% accuracy with XGraphRAG.
  • Efficiency: Debugging speed improved dramatically. Identifying suspicious recalls took approximately 54 seconds with XGraphRAG, compared to 180 seconds with the baseline.
  • Expert Feedback: Participants noted that the tool "restored their confidence" in GraphRAG by making the complex pipeline transparent and manageable.

Limitations and Future Work

While XGraphRAG represents a significant step forward, several areas for improvement remain. The current prototype is primarily tailored to the architecture used by Microsoft’s GraphRAG and assumes a static knowledge graph, meaning it does not yet handle temporal dynamics (e.g., changing roles or relationships over time). Additionally, extremely dense graphs can still lead to visual clutter in the entity view. Future research will focus on more automated "auto-diagnostic" features that can suggest parameter fixes based on identified errors.

Conclusion

XGraphRAG transforms the "black box" of GraphRAG into a transparent, debuggable pipeline. By providing developers with the tools to trace information from the final answer back to the source text, it facilitates the creation of more reliable and accurate graph-based retrieval systems. The project is open-sourced and available at https://github.com/Gk0Wk/XGraphRAG.

Summary updated

Abstract

Graph-based Retrieval-Augmented Generation (RAG) has shown great capability in enhancing Large Language Model (LLM)'s answer with an external knowledge base. Compared to traditional RAG, it introduces a graph as an intermediate representation to capture better structured relational knowledge in the corpus, elevating the precision and comprehensiveness of generation results. However, developers usually face challenges in analyzing the effectiveness of GraphRAG on their dataset due to GraphRAG's complex information processing pipeline and the overwhelming amount of LLM invocations involved during graph construction and query, which limits GraphRAG interpretability and accessibility. This research proposes a visual analysis framework that helps RAG developers identify critical recalls of GraphRAG and trace these recalls through the GraphRAG pipeline. Based on this framework, we develop XGraphRAG, a prototype system incorporating a set of interactive visualizations to facilitate users' analysis process, boosting failure cases collection and improvement opportunities identification. Our evaluation demonstrates the effectiveness and usability of our approach. Our work is open-sourced and available at https://github.com/Gk0Wk/XGraphRAG.
BibTeX
@inproceedings{wang2025xgraphrag,
  author = {Ke Wang and Bo Pan and Yingchaojie Feng and Yuwei Wu and Jieyi Chen and Minfeng Zhu and Wei Chen},
  title = {{XGraphRAG: Interactive Visual Analysis for Graph-based Retrieval-Augmented Generation}},
  booktitle = {PacificVis},
  year = {2025},
  url = {https://openreview.net/forum?id=vGmXKedT5I},
}
Figure