HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation

YiHan Jiao, ZheHao Tan, Dan Yang, DuoLin Sun, Jie Feng, Yue Shen, Jian Wang, Peng Wei

Abstract

Retrieval-Augmented Generation (RAG) significantly enhances Large Language Models (LLMs) by providing external context, yet it often falters when faced with noisy or disparate information. We introduce HIRAG (Hierarchical-Thought Instruction-Tuning), a framework that shifts RAG from a simple "find-and-copy" mechanism to a structured "think-then-answer" process. By training models on a hierarchy of tasks—filtering noise, combining facts, and performing RAG-specific reasoning—HIRAG enables smaller 8B models to outperform much larger counterparts and specialized RAG systems.

Introduction

While LLMs are powerful, they are prone to hallucinations and outdated knowledge. RAG addresses this by allowing models to query external documents. However, current RAG systems suffer from a "garbage in, garbage out" problem: retrieved documents are frequently noisy, irrelevant, or require complex synthesis across multiple sources. Most existing fine-tuning methods lack a structured approach to handle these nuances, often failing to effectively utilize Chain-of-Thought (CoT) reasoning for retrieved text. HIRAG addresses this by instilling a granular, hierarchical set of skills necessary for mastering the "open-book exam" format of modern RAG tasks.

The HIRAG Framework

The core of HIRAG is a hierarchical strategy that introduces three levels of cognitive ability for processing retrieved information:

  1. Filtering: The foundational ability to ignore irrelevant "noise" and pinpoint the specific evidence required.
  2. Combination: The ability to synthesize semantic information across disparate paragraphs or documents to form a cohesive understanding.
  3. RAG-Specific Reasoning: The advanced ability to apply internal logic to external facts, encompassing comparative, deductive, and causal reasoning.
A conceptual comparison between Traditional RAG, which often struggles with information overload and illogical synthesis, and HIRAG's structured hierarchical approach.

Methods and Training Pipeline

HIRAG employs a progressive instruction-tuning approach. The researchers generated a synthetic dataset of 120,000 samples using high-capacity models (e.g., GPT-4o) to produce structured "thoughts." These thoughts are encapsulated within special tokens: <|REASON|> for the internal monologue and <|ANSWER|> for the final output.

To ensure robustness, the training data intentionally includes 20-30% noisy documents. Ablation studies revealed an optimal 1:2:2 ratio for Filtering, Combination, and Reasoning tasks, suggesting that complex synthesis should be prioritized during training. This progressive learning—mastering easier tasks before moving to complex reasoning—ensures better model convergence.

The HIRAG training and inference workflow, illustrating the generation of hierarchical thoughts and final answers across different task levels.

Experimental Results

HIRAG was evaluated across diverse benchmarks, including RGB, PopQA, HotpotQA, and PubMedQA. Key findings include:

  • State-of-the-Art Performance: HIRAG-8B outperformed established RAG models like RAFT and Self-RAG. Notably, it achieved results comparable to or better than Llama-3-70B-Instruct and GPT-4 on several tasks.
  • Multi-hop Reasoning Gains: On the MuSiQue dataset, HIRAG improved accuracy by 7.7 percentage points over previous state-of-the-art models.
  • Noise Robustness: In the RGB-noise benchmark, HIRAG maintained high accuracy even when 80% of the context was irrelevant.
  • Efficiency: The reasoning step is computationally efficient, adding only approximately 0.14 seconds of latency on an A100 GPU for complex tasks.

Conclusion and Future Work

HIRAG demonstrates that teaching models how to think about their references is more effective than simply increasing model size. While it excels at general reasoning, future work will focus on refining performance in highly specialized vertical domains (e.g., legal or technical fields) and exploring reinforcement learning to further diversify reasoning pathways. By structuring the interaction between internal logic and external evidence, HIRAG sets a new standard for robust and intelligent Retrieval-Augmented Generation.

Summary updated

BibTeX
@inproceedings{jiao2025hirag,
  author = {YiHan Jiao and ZheHao Tan and Dan Yang and DuoLin Sun and Jie Feng and Yue Shen and Jian Wang and Peng Wei},
  title = {{HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure