Agentic Multi-Turn Reasoning: A Fairness Approach

Thanh-Dat Truong, Sankalp Pandey, Hugh Churchill, Jackson Cothren, Marios Savvides, Khoa LuuarXiv 2026 · arXiv

TL;DR

$\Phi$-MPO (Fair Multi-Level Preference Optimization) is a new framework designed to train agentic LLMs to handle complex, multi-turn reasoning tasks. By combining fine-grained intermediate state alignment with a focal weighting mechanism, it solves the dual challenges of long-horizon credit assignment and data imbalance, allowing a 7B model to outperform GPT-4o on several reasoning benchmarks.

What Problem Does This Paper Address?

Building "agentic" systems—LLMs that can plan, use tools, and update memory over multiple turns—is notoriously difficult for two main reasons:

  1. Long-Horizon Credit Assignment: In a 10-step reasoning process, you usually only know if the final answer is right or wrong. If the agent fails, was the mistake in the first step (a bad plan), the fifth step (a wrong tool call), or the ninth (a calculation error)? Standard Reinforcement Learning (RL) struggles to distribute "credit" or "blame" to these intermediate steps.
  2. The Imbalance Problem: Training data for reasoning is often highly skewed. Common reasoning patterns (like simple search queries) dominate the dataset, while rare but critical behaviors (like recovering from a tool error or complex logical branching) are underrepresented. Standard optimization tends to ignore these "hard" cases in favor of the "easy" majority.
The influence of data imbalance on performance across different mathematical topics.

Methods

The authors propose Fair Multi-Level Preference Optimization ($\Phi$-MPO), which introduces two key innovations:

  • Multi-Level Preference Learning (MPO): Instead of just comparing two full trajectories, MPO aligns intermediate states between a "preferred" (successful) trajectory and a "dispreferred" (failed) one. It uses semantic similarity (specifically Jaccard overlap of tokenized memory states) to pair up corresponding steps across trajectories of different lengths. This provides a much denser supervision signal.
  • Fair Multi-Level Objective (Focal Weighting): Borrowing an idea from computer vision (Focal Loss), the authors introduce a focal parameter $\gamma$. This mechanism down-weights "easy" preference pairs—where the model already has a high probability of choosing the correct action—and forces the optimization to focus on "hard" pairs where the model is struggling.

The framework also includes a theoretical proof showing that Direct Preference Optimization (DPO) shares the same optimization direction as Group Relative Policy Optimization (GRPO) but is more computationally efficient for agentic rollouts.

Figure 3: Our Proposed Φ-MPO Framework.

Datasets

The model was trained on a mixture of:

  • Search-R1: A dataset focused on multi-turn search and retrieval.
  • DeepMath: A large-scale dataset for mathematical reasoning.

Evaluation was conducted across a broad suite of benchmarks:

  • Knowledge-Intensive Search: Bamboogle, 2Wiki, HotpotQA, and Musique.
  • Agentic Reasoning: GAIA.
  • Mathematical Reasoning: AIME24, AMC23, and GameOf24.
  • Scientific Reasoning: GPQA and MedQA.

Key Insights and Ideas

  • DPO vs. GRPO: The paper provides a mathematical bridge between these two popular methods, proving they are "pairwise equivalent" up to a positive weighting. This justifies using DPO-style objectives for efficiency while maintaining the stability benefits associated with GRPO.
  • Imbalance Propagates: Data imbalance isn't just an input problem; it propagates into the trajectories the agent generates. If an agent rarely sees a specific tool-use case, it will rarely generate it during training, creating a vicious cycle that $\Phi$-MPO's focal weighting is designed to break.
  • Fine-Grained Alignment Matters: Simply knowing a trajectory failed isn't enough. By semantically aligning states (e.g., matching a search step in one trajectory to a search step in another), the model learns why one specific action was better than another at that exact moment.

Key Results

$\Phi$-MPO achieves state-of-the-art (SOTA) performance across the board, often with significantly smaller models:

  • Beating GPT-4o: The $\Phi$-MPO (7B) model outperformed GPT-4o (200B) on Bamboogle (81.6% vs 68.8%) and GAIA (41.7% vs 17.3%).
  • Outperforming Flow-GRPO: On the AIME24 math benchmark, $\Phi$-MPO reached 50.0% accuracy, a significant jump over Flow-GRPO's 40.0%.
  • Scaling Success: When scaled to a Qwen3.5-9B backbone, the performance improved further, reaching 73.3% on AIME24 and 85.6% on Bamboogle.
Comparison of Φ-MPO against GPT-4o and other baselines across multiple benchmarks.

Limitations and Future Work

  • Hyperparameter Sensitivity: The effectiveness of the approach depends on tuning the focal parameter $\gamma$ and the weights between trajectory-level and state-level losses. The authors found $\gamma=2.0$ to be optimal, but this may vary by task.
  • Rollout Quality: The framework relies on "on-the-fly" rollouts to construct preference pairs. If the initial policy is too weak to generate any successful trajectories for a hard task, the model has nothing to learn from.
  • Future Directions: The authors suggest exploring more robust preference optimization strategies that can handle noisy tool feedback or imperfect "LLM-as-judge" rewards, which currently serve as the ground truth for training.

Summary updated

Abstract

Recent advances in Large Language Models (LLMs) have enabled agentic systems capable of solving complex tasks through multi-turn planning, tool use, verification, and memory updates. However, learning agentic systems remains difficult due to two fundamental challenges, i.e., (1) long-horizon credit assignment, where supervision is available only at the final outcome, and (2) imbalanced data distributions, where dominant data patterns bias optimization and weaken adaptation to rare but informative reasoning behaviors. In this paper, we propose Fair Multi-Level Preference Optimization (Fair-MPO or $Φ$-MPO), a new preference optimization framework for agentic learning. We first show that Multi-Level Preference Optimization provides a principled and more computationally efficient framework for long-horizon reasoning. Then, we introduce a Fair Multi-Level Objective that addresses imbalance in agentic learning. We provide a comprehensive theoretical analysis demonstrating that our approach addresses both long-horizon reasoning and data imbalance. Our experiments on agentic reasoning benchmarks demonstrate that our approach achieves State-of-the-Art (SOTA) performance.
BibTeX
@misc{truong2026agentic,
  author = {Thanh-Dat Truong and Sankalp Pandey and Hugh Churchill and Jackson Cothren and Marios Savvides and Khoa Luu},
  title = {{Agentic Multi-Turn Reasoning: A Fairness Approach}},
  year = {2026},
  eprint = {2609.33323},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.33323},
}
Figure