Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning

Wenda Wei, Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Lixin Su, Shuaiqiang Wang, Dawei Yin, Maarten de Rijke, Xueqi Cheng

Abstract

Retrieval-augmented generation (RAG) has proven to be effective in mitigating hallucinations in large language models, yet its effectiveness remains limited in complex, multi-step reasoning scenarios. Recent efforts have incorporated search-based interactions into RAG, enabling iterative reasoning with real-time retrieval. Most approaches rely on outcome-based supervision, offering no explicit guidance for intermediate steps. This often leads to reward hacking and degraded response quality. We propose Bi-RAR, a novel retrieval-augmented reasoning framework that evaluates each intermediate step jointly in both forward and backward directions. To assess the information completeness of each step, we introduce a bidirectional information distance grounded in Kolmogorov complexity, approximated via language model generation probabilities. This quantification measures both how far the current reasoning is from the answer and how well it addresses the question. To optimize reasoning under these bidirectional signals, we adopt a multi-objective reinforcement learning framework with a cascading reward structure that emphasizes early trajectory alignment. Empirical results on seven question answering benchmarks demonstrate that Bi-RAR surpasses previous methods and enables efficient interaction and reasoning with the search engine during training and inference.

Introduction

While Retrieval-Augmented Generation (RAG) helps reduce hallucinations in Large Language Models (LLMs), it often fails in complex, multi-step reasoning tasks. Current state-of-the-art methods like Search-R1 use "outcome-based" reinforcement learning—meaning the model is rewarded only if the final answer is correct. This creates a "black box" for intermediate steps, leading to reward hacking: the model might generate unnecessarily long, rambling, or hallucinated reasoning chains just to stumble upon the right answer. Bi-RAR addresses this by precisely supervising the "information understanding" at every single step of the reasoning process to ensure efficiency and accuracy.

Methodology

The Bi-RAR framework introduces two core innovations to guide the model's "thinking" process:

  1. Bidirectional Information Quantification: Inspired by Kolmogorov complexity, the authors measure the "distance" of each reasoning step. The Step-to-Answer ($d_{T-A}$) measures progress toward the solution, while the Step-to-Question ($d_{T-Q}$) ensures the model stays relevant to the original prompt. These distances are approximated using the LLM's own generation probabilities (log-likelihoods).
  2. Multi-Objective Reinforcement Learning (MORL): The model balances these two objectives using a cascading reward structure that gives higher weight to early correct steps. This encourages the model to find the most efficient path to the answer. The final policy is achieved by training separate forward and backward models using Group Relative Policy Optimization (GRPO) and then combining them via weight-space interpolation.
Comparison of RAG, Search-R1, and Bi-RAR architectures. Bi-RAR incorporates bidirectional feedback loops between reasoning steps and the policy model to ensure both progress toward the answer and relevance to the question.

Experimental Results

The framework was evaluated on seven prominent benchmarks, including single-hop QA (NQ, TriviaQA, PopQA) and multi-hop QA (HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle). Bi-RAR achieved an average relative improvement of 18.2% on the Instruct model over Search-R1, notably using only one-fourth of the training data required by the baseline.

Performance comparison between Search-R1 and Bi-RAR across multiple benchmarks. Bi-RAR consistently produces shorter reasoning chains and requires fewer search engine calls, particularly in complex multi-hop tasks.

Key insights from the results include:

  • Efficiency in Inference: As shown in the figure above, Bi-RAR significantly prunes "rambling" reasoning, resulting in shorter, more dense reasoning chains and fewer search engine calls.
  • Data Efficiency: The bidirectional signals are much more "information-rich," allowing Bi-RAR to outperform state-of-the-art models with 75% less training data.
  • Robustness: The bidirectional design prevents reward fluctuations often seen in outcome-based RL, leading to more stable training trajectories.

Conclusion and Future Work

Bi-RAR demonstrates that evaluating intermediate reasoning steps from both forward and backward directions significantly improves the accuracy and efficiency of retrieval-augmented models. While approximating Kolmogorov complexity via log-probabilities introduces some computational overhead during training, the gains in data efficiency and reasoning quality are substantial. Future work will explore more efficient approximation methods and scaling the framework to larger-parameter models.

Summary updated

Abstract

Retrieval-augmented generation (RAG) has proven to be effective in mitigating hallucinations in large language models, yet its effectiveness remains limited in complex, multi-step reasoning scenarios. Recent efforts have incorporated search-based interactions into RAG, enabling iterative reasoning with real-time retrieval. Most approaches rely on outcome-based supervision, offering no explicit guidance for intermediate steps. This often leads to reward hacking and degraded response quality. We propose Bi-RAR, a novel retrieval-augmented reasoning framework that evaluates each intermediate step jointly in both forward and backward directions. To assess the information completeness of each step, we introduce a bidirectional information distance grounded in Kolmogorov complexity, approximated via language model generation probabilities. This quantification measures both how far the current reasoning is from the answer and how well it addresses the question. To optimize reasoning under these bidirectional signals, we adopt a multi-objective reinforcement learning framework with a cascading reward structure that emphasizes early trajectory alignment. Empirical results on seven question answering benchmarks demonstrate that Bi-RAR surpasses previous methods and enables efficient interaction and reasoning with the search engine during training and inference.
BibTeX
@inproceedings{wei2026thinking,
  author = {Wenda Wei and Yu-An Liu and Ruqing Zhang and Jiafeng Guo and Lixin Su and Shuaiqiang Wang and Dawei Yin and Maarten de Rijke and Xueqi Cheng},
  title = {{Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i40.40675},
}
Figure