Think-on-Graph 2.0: Deep and Interpretable Large Language Model Reasoning with Knowledge Graph-guided Retrieval

Shengjie Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, Jian Guo

Introduction

Large Language Models (LLMs) frequently struggle with complex reasoning tasks requiring deep, specific knowledge, often resulting in hallucinations. While standard Retrieval-Augmented Generation (RAG) mitigates this by fetching text chunks based on semantic similarity, it often fails to capture structural relationships between entities. Conversely, Knowledge Graph (KG)-based RAG utilizes structured "triples" (Subject-Predicate-Object) but is limited by the inherent incompleteness and lack of granularity in KGs.

Think-on-Graph 2.0 (ToG-2) bridges this gap through a hybrid RAG framework that "tightly couples" structured KGs with unstructured text. By iteratively using the KG to guide document retrieval and using document relevance to prune the KG search space, ToG-2 achieves state-of-the-art reasoning performance on complex, multi-hop questions without requiring additional model training.

Methodology: The ToG-2 Framework

ToG-2 operates through an iterative, three-stage process designed to navigate the depth and completeness gap of existing retrieval methods.

The iterative ToG-2 framework showing the coupling of KG search and context retrieval.
  1. Knowledge-Guided Graph Search: The system identifies "topic entities" within a query and explores neighboring entities in the KG. An LLM prunes these relations to maintain focus on the most promising reasoning paths.
  2. Knowledge-Guided Context Retrieval: For every candidate entity identified in the KG, ToG-2 retrieves associated document chunks. Crucially, it leverages the KG relationship (the "triple") to provide context for the retrieval model, ensuring the fetched text is relevant to the specific relationship being explored.
  3. Context-Based Entity Pruning: This "tight coupling" mechanism scores candidate KG entities based on the relevance of their associated text chunks. Only top-scoring entities are retained for the subsequent "hop" of reasoning.

After each round, the LLM evaluates whether it possesses sufficient information to answer. If not, it generates "clues"—summaries of current knowledge and remaining gaps—to guide the next iteration. This process continues until an answer is reached or a maximum depth is hit.

Experimental Evaluation

The authors evaluated ToG-2 across seven knowledge-intensive datasets covering various tasks:

  • Multi-hop KBQA: WebQSP and QALD10-en.
  • Complex Document QA: AdvHotpotQA.
  • Slot Filling: Zero-Shot RE.
  • Fact Verification: FEVER and Creak.
  • ToG-FinQA: A new domain-specific benchmark introduced by the authors, consisting of 2023 Chinese financial statements and 97 manually designed multi-hop questions.

Results and Key Insights

ToG-2 demonstrates superior performance across a wide range of benchmarks, particularly in complex reasoning scenarios.

Performance comparison of ToG-2 against various LLM-only and RAG baselines across multiple datasets.

State-of-the-Art Performance

As shown in the experimental results, ToG-2 achieved state-of-the-art (SOTA) results on 6 out of 7 datasets when using GPT-3.5-turbo as the backbone. Notably, on AdvHotpotQA—which requires navigating complex documents—ToG-2 improved performance by 16.6% over the original ToG.

Domain-Specific Success

On the specialized ToG-FinQA dataset, ToG-2 achieved a 34% Exact Match (EM) score. In contrast, vanilla RAG and Chain-of-Thought (CoT) methods failed completely (0%), highlighting ToG-2's strength in handling specialized, "unseen" data environments where general knowledge is insufficient.

Model Elevation and Efficiency

ToG-2 effectively elevates the performance of smaller, open-source models. For instance, it boosted Llama-3-8B's performance on AdvHotpotQA from 20.8% to 34.7% (a 66.8% relative increase). Furthermore, by using Dense Retrieval Models (DRMs) to prune entities instead of relying solely on expensive LLM calls, ToG-2 reduces the runtime of the entity pruning phase to approximately 68.7% of the original ToG 1.0.

Conclusion

ToG-2 proves that structured knowledge and raw text are more powerful when tightly coupled. The KG provides a roadmap for multi-hop reasoning, while the text provides the granular details necessary to validate reasoning paths. While the system remains dependent on initial KG structure, it represents a significant step toward more interpretable and reliable LLM reasoning. Future work will explore more comprehensive KGs and advanced retrieval "plug-ins" to further optimize the framework.

Summary updated

Abstract

Retrieval-augmented generation (RAG) has enhanced large language models (LLMs) by using knowledge retrieval to address knowledge gaps. However, existing RAG approaches often fail to ensure the depth and completeness of the information retrieved, which is essential for complex reasoning tasks. In this work, we present Think-on-Graph 2.0 (ToG-2), a hybrid RAG framework that iteratively retrieves information from both unstructured and structured knowledge sources in a tightly integrated manner. Specifically, ToG-2 leverages knowledge graphs (KGs) to connect documents via entities, facilitating deep and knowledge-guided context retrieval. Simultaneously, it uses documents as entity contexts to enable precise and efficient graph retrieval. ToG-2 alternates between graph retrieval and context retrieval to search for in-depth clues relevant to the question, enabling LLMs to generate accurate answers. We conduct a series of experiments to demonstrate the following advantages of ToG-2: (1) ToG-2 tightly integrates context retrieval and graph retrieval, enhancing context retrieval through the KG while enabling reliable graph retrieval based on contexts; (2) it achieves deep and faithful reasoning in LLMs through an iterative knowledge retrieval process that integrates contexts and the KG; and (3) ToG-2 is training-free and compatible with various LLMs as a plug-and-play solution. Extensive experiments show that ToG-2 achieves state-of-the-art (SOTA) performance on 6 out of 7 knowledge-intensive datasets with GPT-3.5, and can elevate the performance of smaller models (e.g., LLAMA-2-13B) to the level of GPT-3.5's direct reasoning.
BibTeX
@inproceedings{ma2025think,
  author = {Shengjie Ma and Chengjin Xu and Xuhui Jiang and Muzhi Li and Huaren Qu and Jian Guo},
  title = {{Think-on-Graph 2.0: Deep and Interpretable Large Language Model Reasoning with Knowledge Graph-guided Retrieval}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=oFBu7qaZpS},
}
Figure