Building and maintaining high-quality Knowledge Graphs (KGs) is a constant battle against inconsistency and missing data. While schema validation tools can tell you that something is wrong, they rarely know how to fix it correctly. For instance, if a sensor is missing its connection to a room, is it because the edge was deleted, or because an entire intermediate piece of equipment is missing from the graph?
This paper, "Selective Retrieval Routing for Multi-Violation Knowledge Graph Repair," tackles this ambiguity. The authors propose a multi-agent system where an LLM-based repair agent can "call a researcher" to look up facts in external documents—but only when it’s actually worth the cost.
TL;DR
Blindly using Retrieval-Augmented Generation (RAG) for knowledge graph repair is expensive and can sometimes lead to worse results. This paper introduces a selective routing framework that learns to predict when an LLM should access external evidence versus relying on the graph's internal structure, improving repair accuracy while significantly cutting token costs.
What Problem Does This Paper Address?
In domain-specific KGs (like those for building automation), a "minimal repair"—the one that changes the fewest triples to satisfy a schema—is often factually wrong. To find the intended repair, an agent needs external evidence, such as engineering schedules or floor plans.
However, RAG isn't a free lunch. It consumes massive amounts of tokens and can introduce noise. If the external corpus doesn't contain the answer, or if the retrieval pipeline fails, the LLM might get "distracted" and produce a repair that is less accurate than if it had just looked at the graph's existing labels and structure. The challenge is knowing when to retrieve.
Methods
The authors developed an iterative, multi-agent repair system:
- Iterative Repair Loop: The system validates the graph, clusters related violations together, and repairs them one cluster at a time.
- The Multi-Agent Team:
- Repair Agent: Explores the graph and proposes SPARQL updates to fix violations.
- Researcher Agent: An optional tool that can search a corpus of text, tables, and images to answer the Repair Agent's questions.
- Selective Routing (The "T-learner"): This is the core innovation. Before a repair starts, a router looks at the predicates involved in the violation (e.g.,
observes,hasLocation). It uses two XGBoost models to predict the repair quality ($F_1$ score) both with and without researcher access. It only enables the researcher if the predicted benefit is positive. - Controllable Test Generation: To train this router, the authors extended a framework to "corrupt" valid graphs in controlled ways, creating thousands of training examples with known ground-truth repairs.
Datasets
The study focuses on building automation metadata (ASHRAE Standard 223P), a highly specialized domain where LLMs cannot rely on general knowledge from their pre-training.
- Pritoni Dataset: A prototype building model with a manually constructed corpus of point lists and schematics.
- IBAL Dataset: A real-world laboratory deployment from NIST, featuring authentic engineering drawings and schedules.
Key Insights and Ideas
- Retrieval Utility is Heterogeneous: Retrieval is a massive win for "evidential" predicates (facts typically found in docs, like which sensor measures which room). However, for "representational" predicates (structural "glue" in the graph), retrieval can actually hurt performance by providing irrelevant context that leads the LLM astray.
- Deployment Matters: The same predicate might benefit from retrieval in one building but not another, depending on how well the specific documents cover that information. This justifies the need for a deployment-specific router.
- LLMs are Bad at Self-Routing: When the authors tried using an LLM to decide whether to retrieve (a zero-shot "LLM Router"), it almost always chose to retrieve, failing to be selective and wasting tokens.
Key Results
The selective T-learner router consistently found the "sweet spot" between accuracy and cost:
- Better Accuracy: In 5 out of 6 evaluation suites, the T-learner achieved higher repair $F_1$ scores than the "always OFF" (no retrieval) baseline.
- Lower Cost: It used significantly fewer tokens than the "always ON" (always retrieve) approach.
- Significant Gains: On the IBAL "evidential" suite, enabling the researcher improved the median $F_1$ by +0.357, a massive jump in repair quality.
Limitations and Future Work
- Reference Graph Requirement: Training the router currently requires a "clean" reference graph to generate training cases. In many real-world scenarios, such a graph might not exist.
- Static Routing: The current router is trained offline. If the document corpus changes or the retrieval pipeline is upgraded, the router needs to be retrained. The authors suggest online adaptation (e.g., contextual bandits) as a future direction.
- Black-Box Retrieval: The study treats the "Researcher Agent" as a single unit. Future work could disentangle whether failures are due to poor document coverage or the agent's inability to extract the facts.
Summary updated