Standard Retrieval-Augmented Generation (RAG) operates on a simple premise: if you ask a question, the system should find text snippets that look like that question and show them to a Large Language Model (LLM). This works remarkably well for simple fact-finding, but it hits a wall when faced with "multi-hop" queries—questions that require connecting disparate pieces of evidence to reach a conclusion.
As Boci Peng and colleagues note in Graph Retrieval-Augmented Generation: A Survey, vector-based semantic search is often insufficient for complex queries that require traversing multiple logical steps. When information is distributed across a massive corpus, the "dots" you need to connect might not share any semantic similarity with your initial query, making them invisible to traditional retrievers.
The Semantic Similarity Trap
The fundamental limitation of standard RAG is its reliance on semantic overlap. In a vector database, "The CEO of Company A is Alice" and "Company A acquired Company B" are semantically related to "Company A." However, "Company B’s primary product is a solar panel" might be semantically distant from the original query about "Alice."
Fan Luo and Mihai Surdeanu argue in Divide & Conquer for Entailment-aware Multi-hop Evidence Retrieval that traditional systems fail because they prioritize surface-level similarity over logical entailment. If the evidence needed to answer a question is a chain (A $\rightarrow$ B $\rightarrow$ C), a vector search might find A but completely miss C because C doesn't "look like" the user's question.
Building a Map of Knowledge
To solve this, researchers are moving toward GraphRAG, which structures document collections as a web of interconnected nodes rather than a flat list of chunks.
Different frameworks approach this "map-making" in various ways:
- Associative Memory: BrowseNet transforms documents into a "graph-of-chunks," using named entities to create lexical links that emulate human-like associative memory.
- Hierarchical Tiers: Abdellah Ghassel and Ian Robinson propose the Hierarchical Lexical Graph (HLG), which organizes data into tiers of lineage, summarization, and entity relationships to bridge the gap between semantically distant facts.
- Dual Perspectives: SiReRAG by Nan Zhang and Rui Zhang uses two separate trees—one for semantic similarity and one for entity-based relatedness—recognizing that these two perspectives only overlap by about 50% to 68%.
Navigating the Graph
Once the graph is built, the challenge shifts from finding to navigating. Instead of a one-shot retrieval, these systems often use "agentic" approaches where an LLM acts as a guide, deciding which "hop" to take next.
In HopRAG, Hao Liu and Zhiyu Li describe using LLM-generated "pseudo-queries" as stepping stones. The system finds a document, generates a query that that document might answer, and uses that new query to find the next document in the chain. Similarly, Graph-S3 uses stepwise supervision to reward a retriever for every correct move it makes toward a "golden subgraph" of relevant evidence.
This navigation allows the model to "think on the graph." As Shengjie Ma explains in Think-on-Graph 2.0, this knowledge-guided retrieval ensures that the reasoning chain remains faithful to the underlying data, preventing the model from getting "lost in the noise" of irrelevant but semantically similar distractors.
Overcoming the Token Tax
The primary drawback of GraphRAG is its complexity. Building and querying these graphs can be prohibitively expensive in terms of both computation and LLM tokens. Recent work has focused on making these structures more efficient.
Hong Ting Tsang and Jiaxin Bai introduced TERAG, which replaces expensive LLM-driven graph construction with lightweight concept extraction and Personalized PageRank. This approach achieves high accuracy while using only 3% of the tokens required by state-of-the-art baselines. Similarly, KET-RAG by Xiaokui Xiao uses a "Knowledge Graph Skeleton" for only the most structurally important documents, slashing indexing costs by 90%.
By treating the corpus as a structured landscape to be navigated rather than a bucket of snippets to be searched, GraphRAG enables LLMs to perform the kind of deep, relational reasoning that was previously out of reach for retrieval-augmented systems.