Selective Retrieval Routing for Multi-Violation Knowledge Graph Repair

Tung-Wei Lin, Gabe Fierro, Han Li, Tianzhen Hong, Pierluigi Nuzzo, Alberto Sangiovanni-Vinentelli

Building and maintaining high-quality Knowledge Graphs (KGs) is a constant battle against inconsistency and missing data. While schema validation tools can tell you that something is wrong, they rarely know how to fix it correctly. For instance, if a sensor is missing its connection to a room, is it because the edge was deleted, or because an entire intermediate piece of equipment is missing from the graph?

This paper, "Selective Retrieval Routing for Multi-Violation Knowledge Graph Repair," tackles this ambiguity. The authors propose a multi-agent system where an LLM-based repair agent can "call a researcher" to look up facts in external documents—but only when it’s actually worth the cost.

TL;DR

Blindly using Retrieval-Augmented Generation (RAG) for knowledge graph repair is expensive and can sometimes lead to worse results. This paper introduces a selective routing framework that learns to predict when an LLM should access external evidence versus relying on the graph's internal structure, improving repair accuracy while significantly cutting token costs.

What Problem Does This Paper Address?

In domain-specific KGs (like those for building automation), a "minimal repair"—the one that changes the fewest triples to satisfy a schema—is often factually wrong. To find the intended repair, an agent needs external evidence, such as engineering schedules or floor plans.

Figure 1: A minimal repair satisfies the schema but misses the real-world complexity (a VAV box) found in external documents.

However, RAG isn't a free lunch. It consumes massive amounts of tokens and can introduce noise. If the external corpus doesn't contain the answer, or if the retrieval pipeline fails, the LLM might get "distracted" and produce a repair that is less accurate than if it had just looked at the graph's existing labels and structure. The challenge is knowing when to retrieve.

Methods

The authors developed an iterative, multi-agent repair system:

  1. Iterative Repair Loop: The system validates the graph, clusters related violations together, and repairs them one cluster at a time.
  2. The Multi-Agent Team:
    • Repair Agent: Explores the graph and proposes SPARQL updates to fix violations.
  3. Researcher Agent: An optional tool that can search a corpus of text, tables, and images to answer the Repair Agent's questions.
  4. Selective Routing (The "T-learner"): This is the core innovation. Before a repair starts, a router looks at the predicates involved in the violation (e.g., observes, hasLocation). It uses two XGBoost models to predict the repair quality ($F_1$ score) both with and without researcher access. It only enables the researcher if the predicted benefit is positive.
  5. Controllable Test Generation: To train this router, the authors extended a framework to "corrupt" valid graphs in controlled ways, creating thousands of training examples with known ground-truth repairs.
Figure 2: Overview of the iterative repair system and the routing logic.

Datasets

The study focuses on building automation metadata (ASHRAE Standard 223P), a highly specialized domain where LLMs cannot rely on general knowledge from their pre-training.

  • Pritoni Dataset: A prototype building model with a manually constructed corpus of point lists and schematics.
  • IBAL Dataset: A real-world laboratory deployment from NIST, featuring authentic engineering drawings and schedules.

Key Insights and Ideas

  • Retrieval Utility is Heterogeneous: Retrieval is a massive win for "evidential" predicates (facts typically found in docs, like which sensor measures which room). However, for "representational" predicates (structural "glue" in the graph), retrieval can actually hurt performance by providing irrelevant context that leads the LLM astray.
  • Deployment Matters: The same predicate might benefit from retrieval in one building but not another, depending on how well the specific documents cover that information. This justifies the need for a deployment-specific router.
  • LLMs are Bad at Self-Routing: When the authors tried using an LLM to decide whether to retrieve (a zero-shot "LLM Router"), it almost always chose to retrieve, failing to be selective and wasting tokens.

Key Results

The selective T-learner router consistently found the "sweet spot" between accuracy and cost:

  • Better Accuracy: In 5 out of 6 evaluation suites, the T-learner achieved higher repair $F_1$ scores than the "always OFF" (no retrieval) baseline.
  • Lower Cost: It used significantly fewer tokens than the "always ON" (always retrieve) approach.
  • Significant Gains: On the IBAL "evidential" suite, enabling the researcher improved the median $F_1$ by +0.357, a massive jump in repair quality.
Figure 5: The impact of retrieval varies wildly across different types of violations (Evidential vs. Representational).

Limitations and Future Work

  • Reference Graph Requirement: Training the router currently requires a "clean" reference graph to generate training cases. In many real-world scenarios, such a graph might not exist.
  • Static Routing: The current router is trained offline. If the document corpus changes or the retrieval pipeline is upgraded, the router needs to be retrained. The authors suggest online adaptation (e.g., contextual bandits) as a future direction.
  • Black-Box Retrieval: The study treats the "Researcher Agent" as a single unit. Future work could disentangle whether failures are due to poor document coverage or the agent's inability to extract the facts.

Summary updated

Abstract

Knowledge graphs often contain missing or inconsistent facts that undermine the applications built on them. Schema validation detects such violations but does not identify which of several valid repairs recovers the intended graph. External corpora can supply the evidence needed to identify the intended repair, but retrieving from them is costly and not always successful. We propose a framework for selective retrieval in iterative graph repair. An LLM repair agent explores and edits the graph, while a router controls its access to a researcher agent that retrieves external evidence. We extend a violation-injection framework to generate cases with multiple compatible violations and known reference repairs. The agent's repair outcomes on these cases provide supervision for training the deployment-specific router. On two building-automation datasets, researcher access improves repairs for some violations and degrades them for others, with effects differing across deployments. The router uses fewer repair-time tokens than always enabling access in all six evaluation suites. It achieves higher repair $F_1$ than always disabling access in five of six suites.
BibTeX
@misc{lin2026selective,
  author = {Tung-Wei Lin and Gabe Fierro and Han Li and Tianzhen Hong and Pierluigi Nuzzo and Alberto Sangiovanni-Vinentelli},
  title = {{Selective Retrieval Routing for Multi-Violation Knowledge Graph Repair}},
  year = {2026},
  eprint = {2611.07342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07342},
}
Figure