MMRAG-RFT: Two-stage Reinforcement Fine-tuning for Explainable Multi-modal Retrieval-augmented Generation

Shengwei Zhao, Jingwen Yao, Sitong Wei, Linhai Xu, Yuying Liu, Dong Zhang, Zhiqiang Tian, Shaoyi Du

Introduction

Retrieval-Augmented Generation (RAG) has become a cornerstone for mitigating hallucinations in Large Language Models (LLMs) by providing access to external knowledge. In the multi-modal domain (MMRAG), models must retrieve and synthesize information from both images and text to answer complex queries. However, traditional MMRAG methods often operate as "black boxes," providing answers without clear justification for why specific documents were selected or how they led to a conclusion. Furthermore, the scarcity of high-quality "Chain-of-Thought" (CoT) annotations for multi-modal tasks makes it difficult to train models for effective reasoning using supervised learning alone.

To address these challenges, MMRAG-RFT introduces a two-stage reinforcement learning framework. By leveraging Reinforcement Fine-tuning (RFT) to optimize both document ranking and answer reasoning, the model achieves state-of-the-art results while providing transparent, explainable outputs.

Methodology

The core of MMRAG-RFT is a two-stage process that utilizes the Group Relative Policy Optimization (GRPO) algorithm. This approach allows the model to unlock its inherent reasoning potential without requiring massive human-labeled reasoning chains.

Stage 1: Coarse-Grained Point-Wise Ranking

The first stage involves an efficient filtering pass. The model evaluates candidate documents individually and is trained via rule-based RFT to output a simple "Yes" or "No" regarding a document's relevance. This step effectively removes noise and ensures that the subsequent, more computationally expensive stage focuses only on the most promising evidence.

Stage 2: Fine-Grained List-Wise Ranking and Answering

In the second stage, the model performs joint optimization of ranking and reasoning. It is prompted to generate a structured response that includes a hidden reasoning chain (<think>), the specific IDs of relevant documents (<id>), and the final natural language answer (<answer>). This structure ensures that explainability is built into the model's generation process.

Example of the MMRAG-RFT model generating an explainable response with structured tags.

The Reward Mechanism

Rather than relying on human labels, the model is optimized through a multi-faceted reward system:

  • Format Reward ($R_{format}$): Encourages adherence to the required XML-style tags.
  • Relevance Reward ($R_{match}$): Penalizes the model for selecting incorrect document IDs.
  • Accuracy Reward ($R_{qa}$ and $R_{judge}$): Evaluates the final answer against reference data using metrics like BARTScore.
The Reinforcement Fine-tuning (RFT) framework utilizing Group Relative Policy Optimization (GRPO) and a multi-faceted reward model.

Datasets and Benchmarking

The researchers evaluated MMRAG-RFT on two primary benchmarks:

  • WebQA: A multi-hop, multi-modal dataset requiring retrieval from approximately 40 candidates.
  • MultimodalQA: A diverse dataset covering 16 question types across images, tables, and text.

Additionally, the authors introduced Mini-WebQA, a curated subset of 5,000 samples from WebQA. This new dataset addresses annotation errors and category imbalances (such as an over-representation of color-based questions) found in the original, ensuring more robust and balanced training.

Key Results

MMRAG-RFT established new performance benchmarks across both major datasets:

  • WebQA: Achieved a QA score of 58.3, surpassing the previous state-of-the-art (AETGA) by 4.2 points.
  • MultimodalQA: Reached 71.7% Exact Match (EM) and a 79.7% F1 score, a significant leap over the previous best (RAMQA at 67.0%).

Ablation studies highlighted the importance of the two-stage approach. Removing the Stage 2 reasoning component caused the QA score to drop from 50.9 to 24.4, demonstrating that the joint optimization of ranking and reasoning is essential for high-quality performance.

Discussion and Future Work

A key insight from this research is that Reinforcement Fine-tuning (RFT) significantly outperforms traditional Supervised Fine-tuning (SFT). While SFT can lead to overfitting on specific patterns, RFT allows the model to autonomously "evolve" its reasoning logic, leading to superior generalization. Furthermore, by making the structured <think> format a requirement for maximizing rewards, explainability becomes an intrinsic part of the model's behavior.

Despite these successes, the authors note that generative models still slightly trail specialized discriminative models in pure retrieval precision (the "Retr" score). Future research will focus on scaling this RFT approach to larger models and further bridging the gap between generative reasoning and discriminative retrieval accuracy.

Summary updated

BibTeX
@inproceedings{zhao2026mmrag,
  author = {Shengwei Zhao and Jingwen Yao and Sitong Wei and Linhai Xu and Yuying Liu and Dong Zhang and Zhiqiang Tian and Shaoyi Du},
  title = {{MMRAG-RFT: Two-stage Reinforcement Fine-tuning for Explainable Multi-modal Retrieval-augmented Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i41.40801},
}
Figure