Knowledge-Graph Based RAG System Evaluation Framework

Sicheng Dong, Vahid Zolfaghari, Nenad Petrovic, Alois KnollarXiv 2025 · DOI · arXiv

Introduction

Retrieval-Augmented Generation (RAG) has emerged as the industry standard for grounding Large Language Models (LLMs) in external, verifiable context. By providing models with relevant documents, developers can significantly reduce hallucinations and improve factual accuracy. However, assessing the quality of these systems remains a significant challenge. Traditional metrics like BLEU or ROUGE focus on word overlap rather than semantic meaning, while modern frameworks like RAGAS rely on "atomic facts"—isolated units of information that often fail to capture the complex, relational nature of human knowledge.

Researchers from the Technical University of Munich have proposed a novel evaluation framework that addresses these limitations by representing information as Knowledge Graphs (KGs). By shifting the focus from isolated facts to structural relationships and multi-hop reasoning, this framework provides a more sensitive and nuanced assessment of factual alignment than current industry benchmarks.

The Problem: The Limits of Atomic Facts

Most current RAG evaluation frameworks decompose text into a list of independent statements. While this is a step forward from simple n-gram matching, it ignores the "connective tissue" between facts. In complex reasoning tasks, the relationship between two entities is often as important as the entities themselves. When an evaluator treats facts as isolated atoms, it struggles to verify multi-hop logic—where an answer must be synthesized from multiple disparate pieces of information. This structural deficit can lead to "false positives" in evaluation, where a model receives a high score for mentioning the right entities even if the logical connections between them are incorrect.

Methodology: From Text to Graph

The proposed framework transforms the evaluation process into a graph-theory problem through a three-stage pipeline:

1. Knowledge Graph Construction

Using an LLM, the system extracts "atomic factual triplets" (Subject-Relation-Object) from the input (the question or reference answer) and the context (the retrieved documents or generated response). This process converts unstructured text into a structured network of entities and their specific interactions.

2. Semantic Linking

To account for variations in phrasing (e.g., "authored by" vs. "written by"), the framework employs semantic embedding models (specifically all-MiniLM-L6-v2). It calculates the cosine similarity between nodes; if two entities are semantically equivalent above a predefined threshold, a "SIMILAR" edge is drawn between them, effectively bridging the gap between different linguistic representations of the same concept.

3. Scoring via Graph Algorithms

The framework derives its final scores using two distinct algorithmic approaches:

  • Multi-Hop Semantic Matching: Utilizing a weighted version of Dijkstra’s algorithm, the system checks for the existence of paths between nodes in the input graph and the context graph. This measures the reachability of information, ensuring that the logical chain required to answer a query is present in the retrieved context.
  • Community-Based Semantic Overlap: Using the Louvain community detection algorithm, the global graph is partitioned into thematic clusters. The score is determined by the proportion of these communities that contain nodes from both the input and the context, providing a measure of high-level thematic alignment.

Experimental Validation

The researchers validated the framework using the UDA-QA (Wikipedia-based) and MS MARCO (Bing query-based) datasets. To ensure the automated metrics aligned with human intuition, a subset of the data (10%) was manually annotated by NLP experts across four dimensions: factual correctness, context relevancy, response relevancy, and faithfulness.

Key Findings

  • Superior Sensitivity: In controlled tests, the KG-based metrics proved more "discriminative" than RAGAS. When presented with deliberately incorrect answers, the Multi-Hop Semantic Matching scores dropped to near zero, whereas RAGAS often maintained moderate scores due to directional bias. Conversely, when using ground-truth references, the KG scores consistently approached 1.0.
  • Human Correlation: The framework showed a moderate to high correlation with expert human judgment, particularly regarding factual correctness and answer relevancy.
  • Complementary Strengths: The two algorithms serve different roles: Multi-Hop is best for verifying specific factual logic, while Community-Based Overlap excels at measuring "faithfulness"—ensuring the LLM's response stays within the thematic boundaries of the provided context.

Limitations and Future Directions

The primary challenge facing this framework is scalability. Constructing a Knowledge Graph and running complex graph algorithms is computationally more expensive than simple text-based evaluation. As context windows grow, the time required for triplet extraction and pathfinding increases significantly, which may limit the framework's utility for real-time production monitoring.

Future research will focus on moving from entity-level similarity to triplet-level similarity, optimizing hyperparameters for finer-grained analysis, and exploring "negative rejection"—the ability of an evaluator to confirm when a context correctly does not contain the information required to answer a query.

Summary updated

BibTeX
@misc{dong2025knowledge,
  author = {Sicheng Dong and Vahid Zolfaghari and Nenad Petrovic and Alois Knoll},
  title = {{Knowledge-Graph Based RAG System Evaluation Framework}},
  year = {2025},
  eprint = {2510.02549},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.02549},
}
Figure