TL;DR
This paper introduces the "Tree-of-Question" (ToQ) framework for Korean Question Answering (QA) systems, enhancing retrieval-augmented generation (RAG) by breaking down complex, multi-hop questions into a structured tree. This novel approach, coupled with an improved query generator and a new Korean multi-hop QA dataset, significantly boosts response accuracy and validity, especially for intricate reasoning tasks.
What Problem Does This Paper Address?
Modern QA systems, powered by Large Language Models (LLMs) and Retrieval Augmented Generation (RAG), aim to provide accurate and well-sourced answers. However, handling complex, multi-hop questions – those requiring information from multiple sources or steps of reasoning – remains a significant challenge. This is particularly true for languages like Korean, which have intricate grammatical structures and nuanced word meanings that can complicate query generation. Existing RAG systems often struggle to effectively decompose and plan queries for these complex questions, leading to suboptimal retrieval and inaccurate answers. The paper highlights a critical gap: the lack of established metrics for evaluating the quality of generated queries themselves, which are fundamental to the success of RAG-based QA.
Methods
The core innovation is the Tree-of-Question (ToQ) framework. Inspired by Tree-of-Thought (ToT) prompting, ToQ structures a user's complex question into a tree where each node represents a sub-question. This allows for a more systematic and planned approach to information retrieval.
Key components include:
- Enhanced Query Generator: This component takes the user's question (or a sub-question from the ToQ tree) and transforms it into an optimized query suitable for a search engine. It leverages a few-shot learning approach to adapt to Korean language nuances.
- Answer Integrator: This module precisely identifies and extracts answer spans from retrieved documents that directly match the intent of a given question.
- Query Evaluator: A novel LLM-based evaluator that assesses the semantic coherence and answerability of generated queries and responses. It provides scores for semantic coherence, answerability, overall assessment, and response validity, crucial for evaluating the effectiveness of the query planning process.
The overall RAG process is framed as a sequence of functions: Query Generation, Document Retrieval, and Response Generation. ToQ specifically enhances the Query Generation step by introducing a structured, multi-step decomposition.
Datasets
- Newly Created Korean Multi-hop QA Dataset: The authors developed a new dataset of 200 questions specifically designed for evaluating multi-hop reasoning in Korean. These questions are human-generated and cover various multi-reasoning types (bridging, intersection, comparison, hybrid) as exemplified in Table 1.
- "Naver Cue:" Real-world QA Logs: A subset of 1,000 anonymized questions from the "Naver Cue:" system logs was used for broader evaluation, demonstrating the method's performance on real-world, diverse queries.
Key Insights and Ideas
- Decomposition is Key for Complexity: The ToQ framework demonstrates that breaking down complex questions into a tree of simpler, interconnected sub-questions is far more effective than a single-shot retrieval approach for multi-hop reasoning.
- Korean Language Nuances Matter: The paper emphasizes the challenges posed by Korean's agglutinative structure and subtle word meanings, necessitating specialized query generation techniques.
- Evaluating the Query Planner is Crucial: The authors highlight the critical need for metrics that specifically assess the quality of generated queries, as this directly impacts the entire RAG pipeline. Their Query Evaluator addresses this gap.
- Deeper Reasoning Yields Better Results: Performance, particularly Response Validity, consistently improves as the ToQ tree goes deeper, suggesting that more granular reasoning steps lead to more accurate and relevant answers.
Key Results
- Superior Performance on Multi-hop QA: The ToQ method significantly outperforms baseline methods on the Korean multi-hop QA dataset, achieving a Response Validity of 74.0% compared to other methods.
- Robustness Across Diverse Queries: On a larger dataset of 1,000 random questions, ToQ maintained high performance, achieving a Response Validity of 89.0% in multi-time retrieval scenarios, outperforming baselines like "Previous Context" and "Self-ask."
- Improved Handling of Failures: The ToQ approach successfully addressed 37.3% of single-time retrieval failures at level-1, with increasing success rates at deeper levels.
- Strong Correlation with Human Judgment: The Query Evaluator's metrics, especially Response Validity, show strong positive correlations with human annotations, validating its reliability.
Limitations and Future Work
- Language Scope: The current experiments are limited to Korean. Future work includes extending the approach to English datasets using different LLM backbones to validate its universality.
- Long-tail Questions: The system faces challenges with long-tail questions due to the potential scarcity of relevant documents in search engines, leading to inaccuracies or incomplete answers.
- Inability to Decompose Certain Questions: Approximately 10% of questions are inherently complex and lack a straightforward decomposition path, posing an area for improvement.
- Future Work: The authors plan to extend their experiments to English datasets and continue refining the framework to address the identified limitations.
Summary updated