Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation

Shengjie Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, Cehao Yang, Jiaxin Mao, Jian Guo

TL;DR

Think-on-Graph 2.0 (ToG-2) is a hybrid Retrieval-Augmented Generation (RAG) framework that "tightly couples" structured knowledge graphs (KGs) with unstructured text documents. By iteratively using KGs to guide text retrieval and using retrieved text to prune the graph search, ToG-2 enables large language models (LLMs) to perform deep, multi-hop reasoning that is both more accurate and more faithful to the source material.

What Problem Does This Paper Address?

While Retrieval-Augmented Generation (RAG) has become the standard for grounding LLMs in external facts, it still struggles with complex, multi-step reasoning.

Current approaches generally fall into two camps:

  1. Text-based RAG: Uses vector similarity to find relevant snippets. It’s great for semantic matching but often misses structural links between entities (e.g., connecting a specific athlete to their high school’s records).
  2. KG-based RAG: Uses structured triples (Subject-Predicate-Object) to navigate facts. It provides a clear reasoning roadmap but often lacks the granular detail found in full documents.

Previous "hybrid" attempts usually just combined the results of both (loose coupling). The authors argue that this is insufficient for "deep" reasoning. ToG-2 aims to solve this by making these two sources collaborate: the KG provides the high-level roadmap, while the text provides the granular evidence needed to decide which road to take.

Figure 1: The illustration of differences among (a) text-based RAG, (b) KG-based RAG, (c) KG+Text (Loose-coupling) Hybrid RAG, and (d) our proposed KG×Text (Tight-coupling) RAG framework.

Methods

ToG-2 is a training-free, plug-and-play framework that follows an iterative "Think-on-Graph" loop:

  1. Initialization: The system extracts "topic entities" from the user's question and performs an initial text retrieval to see if the LLM can answer immediately.
  2. Relation Discovery & Pruning: The system explores neighboring relations in the KG. An LLM scores these relations, pruning those unlikely to lead to an answer (e.g., ignoring "place of birth" if the question is about "athletic records").
  3. Knowledge-Guided Context Retrieval: For the remaining candidate entities, ToG-2 retrieves relevant text chunks. Crucially, it doesn't just search for the question; it searches for a combination of the question and the specific KG triple, ensuring the text is contextually linked to the graph path.
  4. Context-Based Entity Pruning: The system uses the relevance of the retrieved text to score and prune the KG entities. This "tight coupling" ensures the graph exploration is grounded in actual document evidence.
  5. Reasoning: The LLM reviews all gathered clues (triples + text). If it has enough info, it answers; if not, it generates new "clues" and starts the next iteration.
Figure 2: An example illustrating the workflow of ToG-2.

Datasets

The authors evaluated ToG-2 on seven knowledge-intensive datasets:

  • Multi-hop KBQA: WebQSP and QALD-10-en.
  • Complex Document QA: AdvHotpotQA (a challenging subset of HotpotQA).
  • Slot Filling: Zero-Shot RE.
  • Fact Verification: FEVER and Creak.
  • New Benchmark: ToG-FinQA. A domain-specific dataset created by the authors consisting of 97 multi-hop questions based on 2023 Chinese financial statements. This serves as a "cold start" test for LLMs that haven't seen this recent data during pre-training.

Key Insights and Ideas

  • The Power of Tight Coupling: The most significant finding is that using text relevance to prune graph nodes (and vice versa) is far more effective than simply aggregating results. It prevents the "lost in the middle" or "hallucination" issues common in long-context RAG.
  • Small Model "Uplift": One of the most impressive "aha moments" was seeing smaller models like Llama-3-8B perform at the level of GPT-3.5-turbo when equipped with ToG-2. This suggests that reasoning capability can be significantly augmented by the structure of the retrieval process, not just the size of the model.
  • Text as the "Meat," KG as the "Skeleton": Manual analysis (Table 4) showed that "Doc-enhanced" answers were the most common (41.94%), confirming that while the KG is essential for navigation, the actual answers usually reside in the unstructured text.

Key Results

ToG-2 achieved state-of-the-art (SOTA) performance on 6 out of 7 datasets using GPT-3.5-turbo.

  • AdvHotpotQA: ToG-2 saw a massive 16.6% improvement over the original ToG (KG-only).
  • ToG-FinQA: In this "unseen data" scenario, standard RAG and Chain-of-Thought (CoT) failed completely (0% Exact Match), while ToG-2 achieved 34%.
  • Efficiency: Despite the extra knowledge sources, ToG-2 is more efficient than its predecessor. By using a "combination strategy" for relation pruning, it reduces LLM API calls from $W$ times per iteration to just once.
Performance comparison across datasets

Limitations and Future Work

  • KG Incompleteness: The framework is still tethered to the quality of the underlying Knowledge Graph. If the KG is missing a critical link, the "roadmap" breaks, even if the information exists in the text documents.
  • Retrieval Model Dependency: The performance is sensitive to the choice of the dense retrieval model (DRM) used for pruning.
  • Evaluation Metrics: The authors noted that Exact Match (EM) often results in "false negatives" where the model provides a correct, nuanced answer that doesn't perfectly match a simplified ground-truth label.
  • Future Directions: The authors suggest exploring more advanced rerankers and expanding the framework to handle even more diverse knowledge sources beyond just Wikipedia and Wikidata.

Summary updated

Abstract

Retrieval-augmented generation (RAG) has improved large language models (LLMs) by using knowledge retrieval to overcome knowledge deficiencies. However, current RAG methods often fall short of ensuring the depth and completeness of retrieved information, which is necessary for complex reasoning tasks. In this work, we introduce Think-on-Graph 2.0 (ToG-2), a hybrid RAG framework that iteratively retrieves information from both unstructured and structured knowledge sources in a tight-coupling manner. Specifically, ToG-2 leverages knowledge graphs (KGs) to link documents via entities, facilitating deep and knowledge-guided context retrieval. Simultaneously, it utilizes documents as entity contexts to achieve precise and efficient graph retrieval. ToG-2 alternates between graph retrieval and context retrieval to search for in-depth clues relevant to the question, enabling LLMs to generate answers. We conduct a series of well-designed experiments to highlight the following advantages of ToG-2: 1) ToG-2 tightly couples the processes of context retrieval and graph retrieval, deepening context retrieval via the KG while enabling reliable graph retrieval based on contexts; 2) it achieves deep and faithful reasoning in LLMs through an iterative knowledge retrieval process of collaboration between contexts and the KG; and 3) ToG-2 is training-free and plug-and-play compatible with various LLMs. Extensive experiments demonstrate that ToG-2 achieves overall state-of-the-art (SOTA) performance on 6 out of 7 knowledge-intensive datasets with GPT-3.5, and can elevate the performance of smaller models (e.g., LLAMA-2-13B) to the level of GPT-3.5's direct reasoning. The source code is available on https://github.com/IDEA-FinAI/ToG-2.
BibTeX
@inproceedings{ma2025think,
  author = {Shengjie Ma and Chengjin Xu and Xuhui Jiang and Muzhi Li and Huaren Qu and Cehao Yang and Jiaxin Mao and Jian Guo},
  title = {{Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=oFBu7qaZpS},
}
Figure