XGraphRAG: Interactive Visual Analysis for Graph-based Retrieval-Augmented Generation

Ke Wang, Bo Pan, Yingchaojie Feng, Yuwei Wu, Jieyi Chen, Minfeng Zhu, Wei Chen

TL;DR

XGraphRAG is a new visual analysis framework designed to help developers debug and optimize Graph-based Retrieval-Augmented Generation (GraphRAG) systems. By providing interactive "inference traces" and multi-level graph visualizations, it allows users to pinpoint exactly where an LLM went wrong—whether it was a failure in entity extraction, a messy knowledge graph merge, or a misleading topic summary.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) has become the standard for grounding LLMs in private or domain-specific data. Recently, GraphRAG has emerged as a powerful evolution, using knowledge graphs to capture complex relationships that standard vector-based RAG misses.

However, GraphRAG is a "black box" on steroids. Its pipeline is incredibly complex: it involves slicing documents, extracting entities/triplets, merging nodes, partitioning the graph into hierarchical topics, and performing multi-stage retrieval. When a GraphRAG system gives a wrong answer, developers are often left "lost in a sea of LLM calls," unable to tell if the error happened during the initial graph construction or the final reasoning step. There is a massive gap in traceability and interpretability.

Methods

The authors propose a two-stage visual analysis workflow implemented in a prototype called XGraphRAG:

  1. Identifying Suspicious Retrievals: The system uses an LLM-assisted framework to reconstruct the "inference path" of both the actual answer and a "ground truth" answer. By comparing these paths side-by-side, it flags Suspicious Recalls—either "Missing Recalls" (facts the LLM should have used but didn't) or "Unexpected Recalls" (irrelevant facts that led the LLM astray).
  2. Analyzing the Root Cause: Once a suspicious recall is found, the user can dive into four specialized views:
  3. QA & Inference Trace View: A side-by-side comparison of reasoning steps.
  4. Topic Explore View: Uses circle packing to show the global hierarchy of the graph, helping users see if the error lies in how topics were summarized.
  5. Entity Explore View: A node-link diagram for local relationship analysis, showing how specific entities are connected.
  6. LLM Invocation View: The "drill-down" tool that shows the exact prompt and output for any specific step in the pipeline (extraction, merging, or summarization).

Datasets

The system was evaluated using the MultiHop-RAG news dataset, which is specifically designed to test an agent's ability to connect multiple pieces of information across different documents. The authors also improved the test questions to ensure they covered both local (entity-specific) and global (topic-wide) queries.

Key Insights and Ideas

  • The "Inference Trace" is King: The most powerful idea here is the reconstruction of the reasoning chain. Instead of just looking at the final answer, developers can see the "evidence facts" the LLM claimed to use and verify them against the actual graph.
  • Local vs. Global Relevance: GraphRAG's strength is its hierarchy. The paper highlights that errors often happen because a "global" topic summary misses a "local" detail. XGraphRAG allows users to jump between these scales seamlessly.
  • Traceability to Source: The system allows a user to click an error in the final answer and trace it all the way back to the specific raw text chunk it was extracted from, revealing if the initial extraction was flawed.

Key Results

The authors conducted a user study with 14 RAG engineers, comparing XGraphRAG against Kotaemon (a popular open-source GraphRAG UI).

  • Accuracy: XGraphRAG significantly improved the success rate of identifying bugs. For complex tasks like finding missing relationships (T4) or faulty topic reports (T5), users had a 0% success rate with the baseline but achieved high accuracy (85%+) with XGraphRAG.
  • Efficiency: Debugging tasks were completed much faster. For example, identifying suspicious recalls (T2) took ~54 seconds with XGraphRAG vs. ~180 seconds with the baseline.
  • Qualitative Feedback: Experts noted that the tool "restored their confidence" in GraphRAG by making the complex pipeline transparent and manageable.

Limitations and Future Work

  • Generality: While the framework is designed to be general, the current prototype is heavily tailored to the architecture used by Microsoft’s GraphRAG.
  • Temporal Dynamics: The current system assumes a static knowledge graph. It doesn't yet handle "streaming" data where relationships change over time (e.g., "Person A was CEO of Company X, but is now at Company Y").
  • Scalability: While the visualizations use hierarchical grouping, extremely dense graphs might still lead to visual clutter ("hairballs") in the entity view.
  • Automation: Future work could involve more automated "auto-diagnostic" features that suggest fixes for the graph construction parameters based on the identified errors.

Summary updated

Abstract

Graph-based Retrieval-Augmented Generation (RAG) has shown great capability in enhancing Large Language Model (LLM)’s answer with an external knowledge base. Compared to traditional RAG, it introduces a graph as an intermediate representation to capture better structured relational knowledge in the corpus, elevating the precision and comprehensiveness of generation results. However, developers usually face challenges in analyzing the effectiveness of GraphRAG on their dataset due to GraphRAG’s complex information processing pipeline and the overwhelming amount of LLM invocations involved during graph construction and query, which limits GraphRAG interpretability and accessibility. This research proposes a visual analysis framework that helps RAG developers identify critical recalls of GraphRAG and trace these recalls through the GraphRAG pipeline. Based on this framework, we develop XGraphRAG, a prototype system incorporating a set of interactive visualizations to facilitate users’ analysis process, boosting failure cases collection and improvement opportunities identification. Our evaluation demonstrates the effectiveness and usability of our approach. Our work is open-sourced and available at https://github.com/Gk0Wk/XGraphRAG.
BibTeX
@inproceedings{wang2025xgraphrag,
  author = {Ke Wang and Bo Pan and Yingchaojie Feng and Yuwei Wu and Jieyi Chen and Minfeng Zhu and Wei Chen},
  title = {{XGraphRAG: Interactive Visual Analysis for Graph-based Retrieval-Augmented Generation}},
  booktitle = {PacificVis},
  year = {2025},
  url = {https://openreview.net/forum?id=vGmXKedT5I},
}

Loading related content...

Figure