Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning

Wenda Wei, Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Lixin Su, Shuaiqiang Wang, Dawei Yin, Maarten de Rijke, Xueqi Cheng

TL;DR

The paper introduces Bi-RAR, a novel retrieval-augmented reasoning framework that prevents large language models (LLMs) from "reward hacking" during multi-step search interactions. By evaluating intermediate reasoning steps in both forward (toward the answer) and backward (grounded in the question) directions using information theory, Bi-RAR achieves state-of-the-art accuracy on QA benchmarks while generating shorter reasoning chains and making fewer search queries.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) helps reduce hallucinations in LLMs, but complex, multi-step questions require the model to iteratively reason and search for new information. Recent advanced paradigms, like Search-R1, train LLMs to interact with search engines using Reinforcement Learning (RL).

However, these methods typically rely on outcome-based supervision—meaning the model is only rewarded if its final answer is correct. This lack of intermediate guidance leads to in-context reward hacking: the model learns to generate excessively long, winding reasoning chains and spams the search engine with loosely relevant queries just to maximize its chances of hitting the right answer. This verbosity degrades efficiency and increases the risk of accumulating hallucinations across reasoning steps.

Bi-RAR addresses this by asking: Can we precisely supervise and guide the information understanding at each intermediate reasoning step?

Comparison of typical RAG, Search-R1, and the proposed Bi-RAR framework.

Methods

Bi-RAR introduces a bidirectional evaluation strategy inspired by human cognition, which utilizes both forward deduction (problem to solution) and backward deduction (solution to problem).

  1. Bidirectional Information Quantification: To evaluate an intermediate step ($T_i$), the authors use Normalized Information Distance (NID) grounded in Kolmogorov complexity theory. Because true Kolmogorov complexity is uncomputable, they approximate it using the log-probabilities of the language model itself. They compute two complementary metrics:
  2. Step-to-Answer Distance ($d_{T-A}$): Measures how much the current step advances toward the final answer (solution progress).
  3. Step-to-Question Distance ($d_{T-Q}$): Measures how well the step remains grounded in the original question (contextual relevance).
Example of how forward and backward information distances are computed for an intermediate step.
  1. Cascading Reward Structure: These distances are converted into rewards using an exponential mapping. To encourage concise reasoning, they apply a cascading factor that diminishes the reward for later steps if early steps have already established the correct direction.
  2. Multi-Objective RL via Weight Interpolation: Simultaneously optimizing for both forward and backward objectives can cause conflicting gradients during early training. To mitigate this, the authors independently train two separate models from the same checkpoint using Group Relative Policy Optimization (GRPO): one optimized for the forward reward ($\theta_{forward}$) and one for the backward reward ($\theta_{backward}$). Finally, they linearly interpolate the weights of the two models ($\theta_{Bi-RAR} = (1-\lambda)\cdot\theta_{forward} + \lambda\cdot\theta_{backward}$) to achieve a balanced, multi-objective solution without extra retraining.

Datasets

The framework was trained on a combined subset of NQ (Natural Questions) and HotpotQA. It was evaluated across seven question-answering benchmarks split into two categories:

  • General QA (Single-hop): NQ, TriviaQA, and PopQA.
  • Multi-hop QA (Complex reasoning): HotpotQA, 2WikiMultiHopQA, Musique, and Bamboogle.

Key Insights and Ideas

  • Cognitive Alignment Works: Forcing the model to look both forward (planning over unknown information) and backward (evaluating known information against the prompt) effectively constrains the reasoning trajectory, preventing the model from drifting into irrelevant tangents.
  • Weight Interpolation Beats Joint Optimization: Blending the weights of two independently trained, single-objective models avoids the optimization instabilities typically found in multi-objective RL training for LLMs.
  • Data Efficiency: Providing fine-grained, step-level feedback drastically reduces the amount of data needed for training. Bi-RAR significantly outperformed the baseline while using only one-fourth of the training data.

Key Results

  • Accuracy Gains: Bi-RAR achieved an average relative performance improvement of 18.2% using Qwen2.5-3B-Instruct and 8.3% using Qwen2.5-3B-Base over the strongest baseline, Search-R1.
  • Inference Efficiency: Bi-RAR successfully curbed model verbosity. As shown in the inference analysis, it generated shorter response lengths and required fewer search engine calls across almost all datasets compared to Search-R1.
Inference efficiency comparison showing response lengths and search calls across datasets.

Limitations and Future Work

  • Computational Overhead: Approximating Kolmogorov complexity via language model generation probabilities requires calculating token-level likelihoods, which can be computationally expensive and time-consuming during training.
  • Model Scaling: The current experiments were restricted to 3B parameter models (Qwen2.5-3B). The authors note that exploring how this framework scales to larger models is an important direction for future work.
  • Approximation Efficiency: Future iterations will focus on finding more efficient mathematical approximations for the bidirectional information distance to speed up training.

Summary updated

Abstract

Retrieval-augmented generation (RAG) has proven to be effective in mitigating hallucinations in large language models, yet its effectiveness remains limited in complex, multi-step reasoning scenarios. Recent efforts have incorporated search-based interactions into RAG, enabling iterative reasoning with real-time retrieval. Most approaches rely on outcome-based supervision, offering no explicit guidance for intermediate steps. This often leads to reward hacking and degraded response quality. We propose Bi-RAR, a novel retrieval-augmented reasoning framework that evaluates each intermediate step jointly in both forward and backward directions. To assess the information completeness of each step, we introduce a bidirectional information distance grounded in Kolmogorov complexity, approximated via language model generation probabilities. This quantification measures both how far the current reasoning is from the answer and how well it addresses the question. To optimize reasoning under these bidirectional signals, we adopt a multi-objective reinforcement learning framework with a cascading reward structure that emphasizes early trajectory alignment. Empirical results on seven question answering benchmarks demonstrate that Bi-RAR surpasses previous methods and enables efficient interaction and reasoning with the search engine during training and inference.
BibTeX
@inproceedings{wei2026thinking,
  author = {Wenda Wei and Yu-An Liu and Ruqing Zhang and Jiafeng Guo and Lixin Su and Shuaiqiang Wang and Dawei Yin and Maarten de Rijke and Xueqi Cheng},
  title = {{Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i40.40675},
}
Figure