TL;DR
This paper introduces a novel "Collaborative Reasoning Framework" (CRF) that combines Reinforcement Learning (RL) with Large Language Models (LLMs) to tackle complex Knowledge Graph Question Answering (KGQA). By mimicking human cognitive processes, CRF leverages LLMs' world knowledge and reasoning capabilities alongside RL's learning from environment to achieve state-of-the-art performance on challenging KGQA tasks.
What Problem Does This Paper Address?
Knowledge Graph Question Answering (KGQA) aims to answer natural language questions by navigating structured knowledge graphs (KGs). While traditional methods struggle with complex questions requiring multi-hop reasoning and constraints, Reinforcement Learning (RL) has shown promise in improving interpretability. However, RL agents often suffer from "aimless exploration" due to a lack of common sense and prior knowledge, leading to slow convergence. Large Language Models (LLMs) possess vast world knowledge but can hallucinate or fail to perform deep, deliberate reasoning. This paper addresses the challenge of effectively answering complex KGQA questions by synergistically combining the strengths of both RL and LLMs, mitigating their individual weaknesses.
Methods
The core of the proposed framework, CRF, is a hierarchical agent that mimics human cognitive processes. It comprises two main components:
- High-level Agent for Constraint Detection: This RL agent identifies constraints or "options" (e.g., basic path extension, filtering, aggregation) encountered during the reasoning process. LLMs provide intermediate rewards by assessing the probability of these options, helping to overcome sparse rewards from the KG environment.
- Low-level Agent for Path Reasoning: This RL agent performs efficient path reasoning within the KG by selecting the most promising relations. It's guided by the high-level agent's chosen option and further enhanced by LLMs, which inject common-sense priors to suggest likely courses of action, preventing aimless exploration.
The framework also uses LLMs as a "guider" to suggest actions and as a "reward function" to provide more informative intermediate rewards. The optimization is performed using the REINFORCE algorithm.
Datasets
The authors evaluated their framework on four widely-used benchmark datasets for KGQA:
- WebQuestionSP (WebQSP): Contains questions with answer entities up to 2 hops away.
- ComplexWebQuestions (CWQ): A more complex dataset extending WebQSP, requiring up to 4 hops and involving four types of constraints.
- PathQuestion (PQ): A general domain dataset with questions extracted from Freebase, focusing on paths between entities.
- MetaQA: A dataset focused on the movie domain, with questions categorized by hop count (1, 2, or 3 hops).
Key Insights and Ideas
- Human Cognition as Inspiration: The framework's hierarchical structure, mimicking System 1 (intuitive) and System 2 (deliberate) thinking, is a key innovation. LLMs act as System 1, providing quick, knowledge-based assessments, while RL acts as System 2, performing structured reasoning and verification.
- LLMs as Common Sense Injectors: LLMs are crucial for guiding the RL agent's exploration, providing common-sense priors that prevent aimless searching and accelerate learning, especially in complex, long-horizon reasoning tasks.
- Collaborative Reward Shaping: Using LLMs to provide intermediate rewards for the high-level agent helps alleviate the problem of sparse and delayed rewards in KGQA, a common issue in RL-based approaches.
- Hierarchical Decision Making for Constraints: Decomposing the KGQA task into constraint detection and path reasoning allows the model to systematically handle complex questions with various constraints.
Key Results
The proposed CRF model achieved state-of-the-art performance on all four benchmark datasets. Notably, it demonstrated significant improvements on challenging datasets like CWQ (9.3% improvement) and WebQSP (3.3% improvement) compared to existing methods. Ablation studies confirmed the effectiveness of each component, particularly the LLM's role in guiding low-level policy and acting as a reward function, highlighting their critical contribution to mitigating aimless exploration and improving performance on complex questions.
Limitations and Future Work
The authors acknowledge that they used a frozen LLM (ChatGPT), and its capabilities might be limited by its pretraining. Future work could explore fine-tuning LLMs to more effectively guide RL agents and improve intermediate reward accuracy. Additionally, the current framework assumes questions contain explicit topic entities. For questions lacking entities, they rely on the LLM to directly generate answers, suggesting future work could integrate this capability more seamlessly.
Summary updated