TL;DR
This paper introduces RoK, a novel framework that enhances Large Language Models (LLMs) for domain-specific question answering by leveraging knowledge graphs (KGs). RoK efficiently selects relevant knowledge paths from KGs to guide LLMs, significantly reducing LLM calls and mitigating hallucination issues, while achieving state-of-the-art performance.
What Problem Does This Paper Address?
Large Language Models (LLMs) like GPT-3.5 and GPT-4 excel at many tasks but struggle with domain-specific question answering due to insufficient domain knowledge and a tendency to "hallucinate" (generate incorrect but plausible-sounding information). Fine-tuning these models can be challenging if they are not open-source or if creating high-quality domain-specific instructions is difficult. While knowledge graphs (KGs) offer structured domain knowledge, existing methods that integrate KGs with LLMs often require numerous LLM calls for reasoning, leading to high computational costs and potential error accumulation from incorrect step selections. Furthermore, these methods often struggle with multi-hop reasoning in KGs and can be inefficient in selecting relevant knowledge paths.
Methods
The proposed RoK framework addresses these challenges through a multi-stage process:
- LLM-driven Reasoning Step Generation: RoK first uses an LLM with Chain-of-Thought (CoT) prompting to generate an initial, step-by-step reasoning process for the question. This process helps to expand the set of key entities related to the question, which is crucial for effective KG integration.
- Knowledge Reasoning Path Generation:
- Main Reasoning Path Selection: RoK identifies candidate paths within the KG connecting the expanded key entities. It then uses PageRank to rank these paths based on entity relevance and the number of key entities they contain, filtering for the most promising paths.
- Neighbor Branch Reasoning Path Selection: To capture implicit information, RoK considers first-order neighbors of entities on the main reasoning path. These neighbor triples are then filtered using an LLM to ensure semantic relevance to the question, with the LLM being called only once for this entire filtering step.
- LLM Answer Generation: The selected main and neighbor reasoning paths are combined to form a comprehensive prompt for the LLM. This prompt provides structured background knowledge and reasoning guidance, enabling the LLM to generate a more accurate and grounded answer with significantly fewer calls compared to previous methods.
Datasets
The authors evaluated RoK on three datasets:
- GenMedGPT-5k: A dataset for medical diagnosis conversations, consisting of 516 QA pairs.
- WebQuestions: An open-domain QA dataset with 400 QA pairs, where answers are expected to be found in Freebase. The authors used a subset, FB15k-237, as the knowledge base.
- CMCQA: A large Chinese medical conversational QA dataset with 400 QA pairs, derived from the Chinese Medical Conversation QA website.
Key Insights and Ideas
- CoT for Entity Expansion is Key: Using Chain-of-Thought prompting not only helps in generating initial answers but crucially expands the set of relevant entities. This richer entity set significantly improves the ability to find relevant paths in the KG, overcoming limitations of relying on just a few directly extracted entities.
- PageRank for Path Prioritization: Applying PageRank to KG paths effectively filters out noisy or less relevant information, ensuring that the most informative reasoning paths are selected. This is a more robust approach than simple retrieval based on semantic similarity.
- Reduced LLM Dependency: By pre-selecting and structuring knowledge paths from KGs, RoK drastically reduces the number of LLM calls required for reasoning. This leads to greater efficiency and lower computational costs.
- Mitigating Hallucinations: The structured knowledge from KGs acts as a strong constraint, guiding the LLM and effectively mitigating its tendency to hallucinate, especially in domain-specific contexts.
Key Results
RoK demonstrated superior performance across the evaluated datasets:
- GenMedGPT-5k & CMCQA: RoK achieved the best GPT-4 average ranking (1.53 and 1.78 respectively), outperforming baselines like GPT-3.5, embedding retrieval, and MindMap. It also showed the highest key entity average match accuracy (81.3% and 91.4% respectively).
- WebQuestions: RoK achieved an accuracy of 80.5%, surpassing other methods.
- Efficiency: RoK requires at least three LLM calls to obtain reasoning paths and high-precision answers, significantly fewer than many existing methods.
Limitations and Future Work
- Limitations: The paper notes that while RoK performs well, the performance is still dependent on the quality and coverage of the knowledge graph. If key entities or relationships are missing from the KG, the system's performance might be affected. The paper also mentions that while RoK reduces LLM calls, it still relies on LLMs for initial reasoning and final answer generation.
- Future Work: The authors suggest that future work could explore more advanced KG reasoning techniques and further optimize the integration of LLMs and KGs. They also imply that extending RoK to other domains beyond medical QA could be a valuable direction.
Summary updated