TL;DR
This paper introduces a novel "Chain of Thought Questioning" (CoT-Questioning) approach for Knowledge Graph Question Answering (KGQA) that breaks down complex questions into smaller, manageable sub-questions for Large Language Models (LLMs). This method, combined with question-type-specific prompting and an ensemble strategy, achieved a first-place F1 score of 85.90 on the TextGraphs-17 Shared Task.
What Problem Does This Paper Address?
Answering complex natural language questions using structured knowledge from Knowledge Graphs (KGs) remains a significant challenge. Traditional KGQA methods, like knowledge retrieval and semantic parsing, struggle with the intricate reasoning required for nuanced questions. While recent work leverages Large Language Models (LLMs) for their reasoning and comprehension abilities, these models can still suffer from factual hallucinations and require effective ways to process complex queries. This paper aims to improve LLM performance on KGQA tasks by enabling them to decompose and reason through complex questions more effectively.
Methods
The core of the NLPeople team's approach is Chain of Thought Questioning (CoT-Questioning). This technique prompts LLMs to break down an initial complex question into a series of simpler sub-questions. By answering these sub-questions sequentially, the LLM can then synthesize the information to arrive at the final answer, mimicking a human's step-by-step reasoning process.
To further enhance performance, the team developed question-type-specific prompting strategies. Recognizing that different question complexities (e.g., ordinal, difference, intersection, superlative) benefit from tailored reasoning, they created few-shot prompts designed for each type. For instance, superlative questions are broken down into factoid questions followed by a comparative operation.
Finally, an ensemble strategy was employed. The outputs from the best-performing models (Llama3-70b-Instruct and GPT 3.5) using CoT-Questioning were combined. If models disagreed, a further entity selection step using GPT 3.5 was used. For unanswered questions, GPT-4 was leveraged.
Datasets
The primary dataset used for this work is from the TextGraphs-17 Shared Task on Text-Graph Representations for KGQA. This dataset is annotated with Wikidata entities and features seven types of complex questions: generic, ordinal, intersection, superlative, difference, multihop, and comparative. The questions themselves originate from the Mintaka dataset, described as a large-scale, complex, and natural language dataset.
Key Insights and Ideas
- Decomposition is Key: Breaking down complex questions into smaller, sequential sub-questions significantly improves LLMs' ability to perform complex reasoning for KGQA.
- Tailored Prompts Matter: Different question types require different reasoning strategies. Question-specific prompts can unlock better performance for challenging question formats.
- Ensembling for Robustness: Combining the strengths of multiple LLMs and prompting techniques can lead to more accurate and robust answers, especially when individual models might falter.
- Wikidata Descriptions Over Graph Strings: Using Wikidata entity descriptions for disambiguation proved more effective than relying solely on linearized graph strings, suggesting that richer contextual information is crucial.
Key Results
The NLPeople team achieved a first-place ranking in the TextGraphs-17 Shared Task with an F1 score of 85.90 on the private test set. This represents a significant improvement, outperforming the baseline GPT 3.5 results by approximately 18%. The paper details performance improvements across various prompting methods and models, with Llama3-70b-Instruct generally showing strong results, especially when combined with CoT-Questioning.
Limitations and Future Work
The authors acknowledge that the effectiveness of CoT-Questioning is dependent on the LLM used, with larger models showing better performance in question decomposition. Additionally, the accuracy of answers can be limited by the training data of the LLM, potentially leading to outdated information. The paper suggests that improved methods for incorporating KG information could further address these limitations. Future work could explore more sophisticated KG integration techniques to mitigate LLM hallucinations and improve the handling of outdated information.
Summary updated