Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization

Xiangchi Yuan, Xingjian Diao, Zheyuan Liu, Chunhui Zhang, Dachuan Shi, Kejing Xia, Zhenbang Du, Shenglong Yao, Soroush Vosoughi, Wenke LeearXiv 2026 · arXiv

TL;DR

Primary Advantage Policy Optimization (PAPO) is a new framework for multi-reward RL that prevents "advantage priority inversion"—a common failure where auxiliary goals (like brevity) accidentally override primary goals (like correctness). By using an asymmetric, sign-preserving fusion rule, PAPO ensures that secondary rewards only refine the policy without ever contradicting the primary objective's preferred direction.

What Problem Does This Paper Address?

When we fine-tune Large Language Models (LLMs) using Reinforcement Learning, we rarely have just one goal. We want the model to be correct (primary), but also concise, safe, or efficient (auxiliary).

The standard industry approach is additive reward aggregation: you simply sum the rewards ($R_{total} = R_{primary} + \alpha R_{auxiliary}$). However, the authors identify a structural flaw called advantage priority inversion. If an auxiliary signal is strong enough and conflicts with the primary one, it can flip the sign of the total advantage. This leads the model to reinforce a wrong-but-short answer over a correct-but-long one.

Existing fixes like "hard gating" (only giving auxiliary rewards if the primary condition is met) are too blunt; they throw away useful comparative data for samples that fail the primary test. PAPO aims to find the "Goldilocks" zone: retaining fine-grained auxiliary supervision while mathematically guaranteeing that the primary objective always keeps its "veto power" over the update direction.

Comparison of additive fusion, hard gating, and the proposed PAPO fusion. PAPO prevents sign inversion while retaining auxiliary distinctions.

Methods

The core of PAPO is a directional asymmetric squashing rule. Instead of a simple sum, it combines advantages ($A_{pri}$ and $A_{aux}$) such that:

  1. If they agree: The signals are added, reinforcing the shared direction.
  2. If they conflict: The auxiliary signal is passed through a tanh function and used to attenuate the primary signal toward zero, but it is mathematically prevented from crossing the zero-axis.

This ensures sign preservation: the final aggregated advantage always shares the sign of the primary advantage.

The authors instantiate this in two popular RL flavors:

  • GRPO-based PAPO: Operates on sequence-level advantages by independently normalizing rewards within a rollout group before fusion.
  • PPO-based PAPO: Operates at the token level. To make this work, they introduce a Dual-Head Critic. Standard critics collapse multiple rewards into one value; PAPO’s critic maintains separate heads for primary and auxiliary returns, allowing for accurate, objective-specific credit assignment before the actor performs the nonlinear fusion.
System overview of PAPO for both GRPO (sequence-level) and PPO (token-level) optimization.

Datasets

The researchers evaluated PAPO across three distinct domains to prove its versatility:

  • Mathematical Reasoning: Trained on DAPO-17K and evaluated on high-stakes benchmarks like OlympiadBench, AIME 2024/2025, and MATH-500. Here, correctness is primary and length is auxiliary.
  • Agentic Tool Use: Used the WHEN2TOOL benchmark, which tests if a model can solve tasks using tools efficiently (success is primary, number of tool calls is auxiliary).
  • Multimodal Safety: Evaluated on BeaverTails-V and MM-SafetyBench, treating safety as the primary goal and helpfulness as the auxiliary one.

Key Insights and Ideas

  • The "Easy Reward" Trap: In standard RL, models often take the path of least resistance by optimizing the "easier" auxiliary reward (like shortening responses) while the "harder" primary reward (correctness) stagnates or even degrades. PAPO breaks this cycle by stabilizing the primary objective's learning curve.
  • Primary-Consistent Ordering: The paper proves that PAPO induces a specific preference structure. Even after the centered normalization used in PPO/GRPO, a response favored by the primary objective will never be ranked below one it disfavors.
  • Token-Level Granularity Matters: In PPO, primary and auxiliary goals might agree on some tokens but conflict on others within the same sentence. By fusing at the token level with a dual-head critic, PAPO resolves these "micro-conflicts" that sequence-level methods miss.
Training dynamics showing how PAPO maintains stable accuracy while progressively improving the auxiliary length reward, compared to baselines that often sacrifice accuracy for brevity.

Key Results

  • Math Superiority: On mathematical reasoning, PAPO-PPO improved average accuracy from 43.6% to 56.6% over standard PPO. In the GRPO family, it raised accuracy from 51.0% to 59.9%.
  • Efficiency without Sacrifice: In tool-use tasks, PAPO achieved the highest accuracy (89.93%) while keeping tool calls low, whereas baselines often reduced tool calls so aggressively that they failed the tasks.
  • Safety Alignment: In multimodal settings, PAPO consistently improved safety scores while maintaining competitive helpfulness, outperforming specialized safety-tuning methods like SafeRLHF-V.
  • Robustness: The method proved effective across different model scales, from 1.5B (DeepSeek-R1-Distill) to 8B (Qwen3) parameters.

Limitations and Future Work

  • Two-Objective Focus: The current formulation focuses on a binary primary-auxiliary relationship. While many tasks fit this mold, extending PAPO to a complex hierarchy of $N$ objectives remains an open research question.
  • Hyperparameter Sensitivity: While PAPO is more stable than additive weighting, it still relies on a coefficient ($\alpha$) to control the strength of auxiliary supervision.
  • Compute Overhead: The dual-head critic in PPO adds a small amount of extra parameters and computation during training, though the authors note this is negligible compared to the overall cost of LLM training.

Summary updated

Abstract

LLM post-training often optimizes multiple objectives with unequal priorities, such as correctness and reasoning efficiency. However, the widely used additive reward aggregation in RL can violate these priorities: when primary and auxiliary advantages conflict, a sufficiently large auxiliary signal can overturn the preference induced by the primary objective. We term this problem \emph{advantage priority inversion}. Although reward-level gating can protect priority more directly, it still removes potentially useful auxiliary distinctions among rollout samples that fail the primary condition. To address this trade-off, we propose Primary Advantage Policy Optimization (\textbf{PAPO}), a priority-aware advantage aggregation method that asymmetrically combines a primary and an auxiliary advantage. PAPO retains auxiliary supervision when the objectives agree and smoothly attenuates conflicting auxiliary signals, preserving the primary sign during fusion for \emph{primary-consistent ordering}. This rule applies to sequence-level advantages in GRPO and token-level advantages in actor--critic PPO. Experiments on multiple domains and models show that PAPO consistently achieves the best primary objective while maintaining best or competitive auxiliary-objective performance across both GRPO- and PPO-based optimization. \href{https://xiangchi-yuan.github.io/papo-project/}{Code is available}.
BibTeX
@misc{yuan2026priority,
  author = {Xiangchi Yuan and Xingjian Diao and Zheyuan Liu and Chunhui Zhang and Dachuan Shi and Kejing Xia and Zhenbang Du and Shenglong Yao and Soroush Vosoughi and Wenke Lee},
  title = {{Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization}},
  year = {2026},
  eprint = {2610.34862},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34862},
}
Figure