Towards Open-World Retrieval-Augmented Generation on Knowledge Graph: A Multi-Agent Collaboration Framework

Jiasheng Xu, Mingda Li, Yongqiang Tang, Peijie Wang, Wensheng ZhangarXiv 2025 · DOI · arXiv

Introduction

Large Language Models (LLMs) have revolutionized natural language processing but continue to struggle with factual hallucinations and a lack of domain-specific, up-to-date knowledge. Retrieval-Augmented Generation (RAG) addresses these issues by grounding model outputs in external data. Knowledge Graphs (KGs) are particularly effective for RAG due to their structured nature and logical relationships.

However, most existing KG-RAG systems operate under a "closed-world" assumption, where the starting "anchor entity" is either provided or easily identified. In real-world scenarios, user queries are often noisy, containing abbreviations, aliases, or typos. This "open-world" setting makes entity linking unreliable. If a system incorrectly identifies the starting node, the subsequent retrieval process is doomed to fail.

Comparison of Closed-World vs. Open-World settings in KG retrieval, illustrating how ambiguous entities like "Romney" can lead to retrieval failure.

The AnchorRAG Framework

To overcome the limitations of single-threaded "link-then-retrieve" methods, we propose AnchorRAG, a Multi-Agent Collaboration (MAC) framework. AnchorRAG decentralizes the retrieval process across three specialized agent roles:

1. The Predictor Agent

The Predictor Agent moves beyond simple string matching. It identifies a set of candidate entities using semantic indexing (FAISS) and employs a "Context-Aware Entity Disambiguation" strategy. By examining the neighboring relations of a node, the agent determines if the local graph structure aligns with the semantic context of the user's question.

2. Retriever Agents

Upon identifying candidate anchors, the framework initializes multiple Retriever Agents to explore the graph in parallel. This "divide and conquer" strategy ensures robustness; if one path leads to a dead end due to an ambiguous anchor, other agents may still find the correct evidence. These agents use LLMs to dynamically score and select the most promising relations and entities to follow.

3. The Supervisor Agent

The Supervisor Agent coordinates the overall process. It synthesizes the paths discovered by the retrievers and validates the evidence. By converting graph paths into natural language, the Supervisor allows the LLM to apply its full reasoning capabilities to verify if the retrieved information truly answers the user's query.

The AnchorRAG multi-agent collaboration workflow featuring Predictor, Retriever, and Supervisor agents working in parallel to resolve a complex query.

For resource-constrained environments, we also introduce AnchorRAG-LR (Lightweight Retrieval). This version replaces expensive LLM-based ranking with a hybrid BM25 and SBERT approach, significantly reducing latency and token costs while maintaining competitive performance.

Experimental Evaluation

AnchorRAG was evaluated on four major Knowledge Graph Question Answering (KGQA) benchmarks: WebQSP, GrailQA, CWQ, and WebQuestions. To test real-world applicability, we developed "Open-World" variants of WebQSP and GrailQA by introducing semantic noise and typos into the queries.

Key Results

  • Performance: AnchorRAG achieved state-of-the-art results, notably improving Hit@1 accuracy on the complex GrailQA dataset by over 20% compared to baselines like Think-on-Graph (ToG).
  • Robustness: In noisy open-world settings, AnchorRAG demonstrated superior resilience. While the ToG baseline saw a 23.4% performance drop on GrailQA, AnchorRAG's performance decreased by only 11.6%.
  • Efficiency: The AnchorRAG-LR variant reduced token consumption by 77.8% and reasoning time by 63.8% compared to the full model, yet still outperformed existing strong baselines.

Conclusion

AnchorRAG represents a significant step toward practical, open-world KG-RAG. By leveraging multi-agent parallelism and context-aware disambiguation, the framework mitigates the "single point of failure" in entity linking. Future research will focus on refining agent communication protocols and scaling the framework to massive, heterogeneous data structures beyond current benchmarks.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in web search and reasoning. However, their dependence on static training corpora makes them prone to factual errors and knowledge gaps. Retrieval-Augmented Generation (RAG) addresses this limitation by incorporating external knowledge sources, especially structured Knowledge Graphs (KGs), which provide explicit semantics and efficient retrieval. Existing KG-based RAG approaches, however, generally assume that anchor entities are accessible to initiate graph traversal, which limits their robustness in open-world settings where accurate linking between the user query and the KG entity is unreliable. To overcome this limitation, we propose AnchorRAG, a novel multi-agent collaboration framework for open-world RAG without the predefined anchor entities. Specifically, a predictor agent dynamically identifies candidate anchor entities by aligning user query terms with KG nodes and initializes independent retriever agents to conduct parallel multi-hop explorations from each candidate. Then a supervisor agent formulates the iterative retrieval strategy for these retriever agents and synthesizes the resulting knowledge paths to generate the final answer. This multi-agent collaboration framework improves retrieval robustness and mitigates the impact of ambiguous or erroneous anchors. Extensive experiments on four public benchmarks demonstrate that AnchorRAG significantly outperforms existing baselines and establishes new state-of-the-art results on the real-world reasoning tasks.
BibTeX
@misc{xu2025towards,
  author = {Jiasheng Xu and Mingda Li and Yongqiang Tang and Peijie Wang and Wensheng Zhang},
  title = {{Towards Open-World Retrieval-Augmented Generation on Knowledge Graph: A Multi-Agent Collaboration Framework}},
  year = {2025},
  eprint = {2509.01238},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.01238},
}
Figure