KnowGPT: Knowledge Graph based Prompting for Large Language Models

Qinggang Zhang, Junnan Dong, Hao Chen, Daochen Zha, Zailiang Yu, Xiao Huang

TL;DR

KnowGPT is a novel framework that connects closed-source, black-box Large Language Models (LLMs) with structured Knowledge Graphs (KGs) using optimized "hard" prompts. By combining deep reinforcement learning for concise knowledge extraction and multi-armed bandits for adaptive prompt construction, KnowGPT drastically reduces hallucinations and API costs, even outperforming zero-shot GPT-4 on complex question-answering tasks.

What Problem Does This Paper Address?

While Large Language Models (LLMs) are incredibly capable, they frequently suffer from "hallucinations"—fabricating plausible-sounding but incorrect facts when queried on domain-specific or niche topics. Retrieval-Augmented Generation (RAG) attempts to solve this by fetching text documents, but unstructured text can be noisy, incomplete, or contradictory.

Knowledge Graphs (KGs) offer a cleaner alternative by storing explicit, structured factual triples (e.g., [Entity A] -> [Relation] -> [Entity B]). However, integrating KGs into state-of-the-art closed-source models (like GPT-4 or Claude 3) via APIs introduces three major challenges:

  1. Huge Search Space: Real-world KGs contain millions of triples, making it difficult to find the exact facts relevant to a query.
  2. High API Costs: Dumping massive subgraphs into an LLM prompt explodes token counts and incurs heavy financial costs.
  3. Laborious Prompt Engineering: LLMs are highly sensitive to prompt formatting, and a rigid, manually designed template rarely fits every type of question.
Figure 1: A real-world question from Open- bookQA. GPT-3.5 could effectively correct the answer given the scientific reasoning background from ConceptNet (blue: question concepts, red: answers, grey: entities not present in questions).

Methods

KnowGPT introduces a two-stage pipeline to efficiently bridge the gap between KGs and black-box LLMs:

1. Knowledge Extraction via Deep Reinforcement Learning (RL)

Instead of retrieving a massive, noisy neighborhood of graph nodes, KnowGPT treats knowledge retrieval as a path-finding problem. It trains a reinforcement learning agent (using a policy gradient network) to walk the KG from source entities to target entities. The agent is guided by a custom reward function that balances three criteria:

  • Reachability: Did the path successfully connect the question to a candidate answer?
  • Context-Relatedness: Is the semantic embedding of the path close to the embedding of the question context?
  • Conciseness: Is the path short? The model penalizes long, rambling chains to keep prompt lengths minimal.

2. Prompt Construction via Multi-Armed Bandits (MAB)

Once the concise paths are extracted, they must be translated into text. KnowGPT supports three prompt formats: raw Triples ($F_t$), colloquial Sentences ($F_s$), and a high-level Graph Description ($F_g$) generated by an LLM.

Because different questions require different formats, KnowGPT employs a Multi-Armed Bandit (MAB) algorithm. The MAB acts as a meta-selector, dynamically learning which combination of knowledge extraction and prompt format yields the highest accuracy for a specific question type, smoothly balancing exploration and exploitation.

Figure 2: The overall architecture of our proposed knowledge graph prompting framework, i.e., KnowGPT. Given the question context with multiple choices, we first retrieve a question-specific subgraph from the real-world KG. Knowledge Extraction is first dedicated to searching for the most informative and concise reasoning background subject to the context. Then the Prompt Construction module is optimized to prioritize the combination of knowledge and formats subject to the given question.

Datasets

The authors evaluated KnowGPT across three benchmark datasets spanning general commonsense and specialized domains:

  • CommonsenseQA (CSQA): A 5-option multiple-choice dataset requiring everyday reasoning, paired with the ConceptNet knowledge graph.
  • OpenBookQA (OBQA): A 4-option dataset testing elementary scientific facts, also evaluated using ConceptNet.
  • MedQA-USMLE: A highly complex biomedical dataset sourced from official US Medical Licensing Examinations, paired with a specialized USMLE knowledge graph (amalgamating UMLS and DrugBank).

Key Insights and Ideas

  • Conciseness Directly Lowers API Overhead: By optimizing for path conciseness during the RL phase, KnowGPT filters out redundant graph data. This proves that smart filtering is better than throwing raw context volume at an LLM.
  • Prompt Formats Must Match Question Complexity: A fascinating takeaway from the ablation studies is that simple questions perform best with raw triples ($F_t$). However, complex, multi-hop, or graph-heavy reasoning questions see massive performance spikes when using the richer "Graph Description" ($F_g$) format.
  • Black-Box Prompting Can Beat White-Box Fine-Tuning: KnowGPT (built on top of a frozen GPT-3.5) outperformed open-source models that were explicitly fine-tuned with graph embeddings, demonstrating the sheer power of optimized prompt spaces.

Key Results

KnowGPT achieved state-of-the-art performance across the board, consistently outperforming its baseline (GPT-3.5) and even surpassing zero-shot GPT-4:

  • Accuracy Boost: KnowGPT achieved an average accuracy improvement of 23.7% over GPT-3.5 and 2.9% over GPT-4 across the three datasets.
  • Leaderboard Success: On the official OpenBookQA leaderboard, KnowGPT achieved 92.6% accuracy, outperforming ensemble methods and approaching human-level performance (91.7%).
  • Cost Efficiency: On the MedQA dataset, KnowGPT averaged just 348 tokens per prompt (costing \$6.64), compared to competing frameworks like Mindmap (761 tokens, \$14.52) and Chain-of-Knowledge (1129 tokens, \$21.54), making it highly practical for real-world deployment.
Table 1: Performance comparison among baseline models and KnowGPT on three benchmark datasets.

Limitations and Future Work

The primary limitation of KnowGPT is its dependency on the quality of the underlying Knowledge Graph. Real-world KGs are often constructed automatically from web crawls and inevitably contain noisy, incorrect, or outdated triples. If the RL agent extracts a noisy path, it can mislead the LLM into a wrong prediction. The authors note that future iterations will explore integrating off-the-shelf KG refinement and denoising algorithms prior to the knowledge extraction phase.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in many real-world applications. Nonetheless, LLMs are often criticized for their tendency to produce hallucinations, wherein the models fabricate incorrect statements on tasks beyond their knowledge and perception. To alleviate this issue, researchers have explored leveraging the factual knowledge in knowledge graphs (KGs) to ground the LLM's responses in established facts and principles. However, most state-of-the-art LLMs are closed-source, making it challenging to develop a prompting framework that can efficiently and effectively integrate KGs into LLMs with hard prompts only. Generally, existing KG-enhanced LLMs usually suffer from three critical issues, including huge search space, high API costs, and laborious prompt engineering, that impede their widespread application in practice. To this end, we introduce a novel Knowledge Graph based PrompTing framework, namely KnowGPT, to enhance LLMs with domain knowledge. KnowGPT contains a knowledge extraction module to extract the most informative knowledge from KGs, and a context-aware prompt construction module to automatically convert extracted knowledge into effective prompts. Experiments on three benchmarks demonstrate that KnowGPT significantly outperforms all competitors. Notably, KnowGPT achieves a 92.6% accuracy on OpenbookQA leaderboard, comparable to human-level performance.
BibTeX
@inproceedings{zhang2024knowgpt,
  author = {Qinggang Zhang and Junnan Dong and Hao Chen and Daochen Zha and Zailiang Yu and Xiao Huang},
  title = {{KnowGPT: Knowledge Graph based Prompting for Large Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=PacBluO5m7},
}
Figure