TL;DR
This paper introduces RDR2, a novel framework that equips Retrieval-Augmented Generation (RAG) models with an understanding of document structure. By dynamically navigating document hierarchies, RDR2 significantly improves the ability of LLMs to acquire and utilize knowledge, especially in complex multi-document scenarios, achieving state-of-the-art results.
What Problem Does This Paper Address?
Large Language Models (LLMs) often struggle with factual inaccuracies due to their reliance on parametric knowledge. Retrieval-Augmented Generation (RAG) aims to mitigate this by retrieving information from external documents. However, existing RAG approaches treat retrieved passages as isolated chunks, ignoring the inherent structure within documents (like headings, sections, and their hierarchical relationships). This structural blindness prevents RAG systems from fully leveraging how humans naturally navigate and reason about information, leading to suboptimal knowledge acquisition and utilization, particularly in complex tasks requiring synthesis from multiple sources.
Methods
The core innovation is the Retrieve-DocumentRoute-Read (RDR2) framework, which explicitly incorporates document structure throughout the RAG process.
- Document Structure Representation: Documents are represented as Document Structure Trees (DSTs), encoding hierarchical relationships with "Structure Nodes" (headings) and "Content Nodes" (passages). A Retrieval SubTree (RST) is derived from the DST, maintaining a stable retrieval scope while adaptively updating contextual focus.
- LLM-based Router: RDR2 employs a trainable LLM-based router that dynamically navigates the document structure tree. This router performs three atomic actions:
- [ANS]: Selects a content node that directly answers the question.
- [EXP]: Unfolds a collapsed structure node if its heading or context suggests relevance.
- [REF]: Stops exploring a subtree when no relevant nodes are found.
- Action Curation: A novel method automatically curates routing actions from questions alone, without needing answer supervision, by retrieving passages and then prompting an LLM to generate routing actions based on the document structure.
- Training: The router is fine-tuned using a supervised approach on the curated routing actions.
Datasets
The framework was evaluated on five challenging knowledge-intensive QA datasets:
- TriviaQA: Short-form, single-answer trivia questions.
- HotpotQA: Multi-hop QA requiring interleave retrieval and reasoning.
- QAMPARI: List-style QA with multiple factual short entities.
- ASQA: Long-form, ambiguous QA requiring coherent synthesis.
- ELI5: Complex, open-ended questions requiring in-depth explanations.
The routing model was trained on a subset of ASQA questions.
Key Insights and Ideas
- Structure is Key for Complex RAG: The paper strongly suggests that understanding document structure is not just a minor enhancement but a critical factor for improving RAG performance, especially in complex, multi-document synthesis tasks.
- Human-like Navigation: RDR2 mimics human reading strategies by dynamically exploring document hierarchies, which leads to more efficient and effective knowledge retrieval.
- Trainable Routing: Formulating document routing as a trainable task, rather than a fixed heuristic, allows the model to learn optimal navigation strategies.
- Generalization Power: The RDR2 framework demonstrates strong generalization capabilities, performing well even on datasets where the router was not explicitly trained.
- Efficiency Gains: Despite incorporating structural awareness, RDR2 achieves competitive or better results with significantly shorter outputs compared to some baselines, indicating improved efficiency in information delivery.
Key Results
- State-of-the-Art Performance: RDR2 achieves state-of-the-art results across multiple challenging datasets (QAMPARI, ASQA, ELI5).
- Significant Improvements: For example, on ASQA, RDR2 achieved a +10.8% EM improvement over standard RAG with minimal computational overhead. On QAMPARI, it showed a +21.1% EM gain.
- Robustness: The framework demonstrates robustness to different readers and retrievers, validating the effectiveness of its structural awareness.
- Test-time Scaling: RDR2 allows for dynamic test-time compute scaling without model weight updates, offering flexibility for applications with varying latency requirements.
Limitations and Future Work
- Offline DST Construction: The current framework requires offline construction of Document Structure Trees (DSTs), which can take time (e.g., 20 minutes for Wikipedia with parallelization).
- Intra-document Focus: The routing mechanism primarily navigates intra-document hierarchies. Integrating knowledge across multiple documents more effectively remains an area for future work.
- Controlled Expansion: While RDR2 offers tunable trade-offs between performance and cost via expansion iterations, further optimization of this mechanism could be explored.
Summary updated