RJE: A Retrieval-Judgment-Exploration Framework for Efficient Knowledge Graph Question Answering with LLMs

Can Lin, Zhengwang Jiang, Ling Zheng, Qi Zhao, Yuhang Zhang, Qi Song, Wangqiu ZhouEMNLP 2025

TL;DR

This paper introduces RJE, a novel framework that makes Large Language Models (LLMs) more efficient and accurate at answering questions using knowledge graphs (KGs). RJE intelligently retrieves relevant information, judges its sufficiency, and conditionally explores further, significantly reducing computational costs while maintaining high performance, even with smaller LLMs.

What Problem Does This Paper Address?

Answering complex questions using knowledge graphs (KGQA) is a challenging task. While LLMs have shown promise, existing methods face limitations. Retrieval-based methods are often constrained by the quality of the information they pull, leading to either incomplete answers or noisy, irrelevant data. Agent-based methods, on the other hand, rely heavily on powerful, proprietary LLMs and can be computationally expensive. This creates a need for KGQA systems that are both accurate and efficient, especially when using smaller, open-source LLMs.

Methods

The core of RJE is a three-stage framework:

  1. Retrieval: This stage focuses on finding relevant reasoning paths within the KG. It first retrieves candidate paths based on relevance to the question and then uses a lightweight "Reasoning Path Ranking" module to prioritize the most promising ones. This ensures that the LLM receives high-quality, relevant information from the start.
  2. Judgment: Here, an LLM acts as a judge. It evaluates whether the retrieved reasoning paths provide enough information to answer the question directly. If sufficient, the LLM generates the answer immediately, saving computational resources.
  3. Exploration: If the judgment stage finds the information insufficient, RJE enters an "Exploration" phase. This is where RJE innovates beyond prior agent-based methods. It first uses "Question Decomposition" to break down the original question into smaller, entity-focused sub-questions. Then, it employs "Retriever-assisted Exploration," where the retriever pre-filters candidate relations to guide the LLM's search, making the exploration more targeted and efficient. This avoids the LLM having to explore blindly.

RJE also introduces specialized auxiliary modules: Reasoning Path Ranking, Question Decomposition, and Retriever-assisted Exploration. These modules are designed to reduce the reasoning burden on LLMs, making them more effective, especially smaller ones.

Datasets

The authors evaluated RJE on two standard KGQA benchmark datasets:

  • WebQuestionsSP (WebQSP): Based on the Freebase KG.
  • Complex WebQuestions (CWQ): Also based on the Freebase KG.

Key Insights and Ideas

  • Intelligent Sufficiency Check: The "Judgment" stage is crucial. By having an LLM first assess if the retrieved information is sufficient, RJE avoids unnecessary exploration, leading to significant efficiency gains.
  • Targeted Exploration: Instead of LLMs exploring KGs broadly, RJE's "Retriever-assisted Exploration" and "Question Decomposition" guide the LLM to focus on specific, relevant parts of the KG, drastically reducing search space and computational cost.
  • Empowering Smaller LLMs: The framework's design, particularly the auxiliary modules, allows smaller open-source LLMs to achieve performance comparable to much larger, proprietary models, democratizing access to powerful KGQA capabilities.
  • Balancing Retrieval and Reasoning: RJE strikes a balance by retrieving relevant paths and then using LLMs for judgment and targeted exploration, rather than relying solely on one or the other.

Key Results

  • Superior Accuracy: RJE consistently outperforms existing state-of-the-art methods on both CWQ and WebQSP datasets, even when using proprietary LLMs like GPT-40-mini.
  • Efficiency Gains: RJE significantly reduces LLM calls, token usage, and inference time compared to methods like ToG and PoG. For example, on CWQ, RJE reduces LLM calls by up to 65% and is up to 5.9 times faster.
  • Small LLM Performance: RJE enables small open-source LLMs (e.g., Llama3.2-3B) to achieve performance comparable to larger models. For instance, RJE with Llama3.2-3B achieved a 41.5% improvement over PoG on CWQ for the same model size.
  • Component Effectiveness: Ablation studies show that each stage and auxiliary module of RJE contributes positively to overall performance, highlighting the synergistic design.

Limitations and Future Work

  • KG Quality: The performance of RJE, like any KGQA system, is dependent on the quality of the underlying KG. Noisy or outdated triples can still mislead LLMs. The authors plan to investigate methods for detecting and filtering unreliable knowledge.
  • Language Support: The current evaluation is limited to English language datasets. Future work will involve extending the evaluation to multiple languages to assess cross-lingual capabilities.

Summary updated

BibTeX
@inproceedings{lin2025rje,
  author = {Can Lin and Zhengwang Jiang and Ling Zheng and Qi Zhao and Yuhang Zhang and Qi Song and Wangqiu Zhou},
  title = {{RJE: A Retrieval-Judgment-Exploration Framework for Efficient Knowledge Graph Question Answering with LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure