Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization

Yang Bai, Kaiyuan Liu, Ziyuan Zhuang, Jiahong Zhou, Rongxiang Weng, Xin Chen, Jingang Wang, Xunliang CaiarXiv 2026 · arXiv

TL;DR

Training Large Language Models (LLMs) using Reinforcement Learning (RL) often requires balancing multiple, conflicting reward signals (like correctness, formatting, and length), which frequently causes training instability. This paper introduces Reward-Decorrelated Policy Optimization (RDPO), a lightweight pipeline that uses quantile mapping to normalize diverse reward scales and Mahalanobis whitening to eliminate redundant correlations. When applied to the post-training of the LongCat-Flash model, RDPO significantly boosts instruction-following and writing quality while maintaining top-tier reasoning and coding performance.

What Problem Does This Paper Address?

In modern LLM alignment, a single training run usually juggles multiple objectives simultaneously. For example, a model needs to generate text that is factually accurate, adheres to complex formatting constraints, follows stylistic rubrics, and remains concise.

When we feed these diverse signals into standard RL frameworks like GRPO, we hit two major roadblocks:

  1. Scale and Distribution Mismatch: A binary pass/fail reward (e.g., code execution) behaves entirely differently from a continuous preference score or a quadratic length penalty. Standard linear normalization (Z-scoring) is highly sensitive to outliers and skewed distributions, often causing a single anomalous rollout to hijack the entire gradient update for a prompt.
  2. Inter-Reward Correlation: Many reward dimensions are inherently correlated. For instance, longer responses might naturally score higher on a general preference model, or thorough mathematical reasoning might require more tokens. Simply summing these rewards double-counts redundant information or inadvertently cancels out useful variance.

Methods

RDPO mitigates these issues through a lightweight, two-step reward processing pipeline executed before rewards are aggregated into a scalar advantage:

1. Magnitude-Aware Quantile (MAQ) Normalization

Instead of relying on standard Z-scores, RDPO maps rewards to a standard normal distribution based on their relative ranks within a prompt's rollout group. Crucially, to avoid losing the actual physical differences between scores (which happens in pure rank transformation), MAQ introduces log-compressed magnitude gaps. This preserves local quantitative nuances while compressing extreme outliers, ensuring stable advantage allocation across all rollouts.

2. Subspace Mahalanobis Whitening

To eliminate cross-reward redundancy, RDPO maintains a running estimate of the reward covariance matrix using an Exponential Moving Average (EMA). It then applies Mahalanobis whitening to decorrelate the dimensions, effectively transforming them into orthogonal, unit-variance signals.

Because different tasks activate different subsets of rewards (e.g., math tasks don't use formatting rubrics), RDPO applies this whitening dynamically only within the active reward subspace of the given task. This prevents the model from calculating artificial correlations between rewards that never co-occur.

Datasets

The authors validated RDPO during the post-training phase of the LongCat-Flash model using a rich mixture of four task categories: mathematical reasoning, code generation, instruction following, and general writing.

To evaluate the model, they leveraged a comprehensive suite of established benchmarks:

  • Instruction Following: IFEval, GuideBench, SOP-Maze
  • Math & Knowledge Reasoning: AIME24, AIME25, GPQA, MATH500
  • Writing & Arena Evaluation: WritingBench, ArenaHard-v2 (including AH-Hard and AH-Creative subsets)
  • Coding: FullStackBench, HumanEval+, MBPP+, LiveCodeBench v6

Key Insights and Ideas

  • The Danger of Advantage Domination: The authors show that under standard Z-score normalization (GDPO), the advantage mass frequently concentrates on a single rollout. MAQ successfully counteracts this, lowering advantage domination and boosting effective rollout participation across all task types.
MAQ stabilizes prompt-level advantage allocation compared to Z-score normalization.
  • Measuring "Information Efficiency": The paper introduces a novel diagnostic metric, Effective Information Efficiency ($\eta_{eff}$), which quantifies whether an aggregation method balances weights evenly and avoids redundant variance. RDPO maintains a substantially higher and more stable information efficiency throughout the entire training trajectory compared to baselines.
Effective Information Efficiency ($\eta_{eff}$) across training steps.
  • Subspace-Specific Processing is Essential: In multi-task learning, a global whitening matrix fails because reward dimensions don't always overlap. Dynamically isolating the active reward subspace ensures that decorrelation only happens where it mathematically makes sense.
RDPO dramatically reduces absolute Pearson correlations within active task subspaces.

Key Results

The authors first performed a small-scale ablation study on a smaller model family, demonstrating that combining MAQ and Whitening (Q+M) yields a compounding benefit (e.g., IFEval scores jumped from a baseline of 78.56 to 83.55).

When scaled up to the full LongCat-Flash model, RDPO achieved standout results:

  • Instruction Following & Writing: RDPO achieved 90.39% on IFEval (vs. GRPO's 89.46%) and 87.63% on WritingBench (vs. GRPO's 84.12%).
  • Hard Prompts: On the challenging ArenaHard-v2 (Hard) benchmark, RDPO surged to 76.10%, a massive leap over GRPO (65.80%) and the initialization baseline (49.40%).
  • Reasoning & Coding Consistency: While optimizing heavily for alignment and style constraints usually degrades core capabilities, RDPO remained highly competitive, matching or slightly exceeding GRPO on MATH500 (98.60%) and LiveCodeBench (63.80%).

Limitations and Future Work

  • Covariance Warmup Dependency: The Mahalanobis whitening step relies on an accurate running covariance matrix. At the very beginning of training, this estimate is unstable, requiring a manual "warmup phase" (5 steps in this work) before whitening can be safely applied.
  • Heuristic Gating for Core Constraints: To prevent auxiliary rewards (like length penalties) from overriding core task success, the authors still had to implement manual conditional rules (e.g., truncating preference rewards if a core rubric fails). Natively incorporating these hierarchical constraints directly into the decorrelation mathematics remains an open area for future research.

Summary updated

Abstract

Complex reinforcement learning environments frequently employ multi-task and mixed-reward formulations. In these settings, heterogeneous reward distributions and correlated reward dimensions often destabilize the construction of scalar advantages. To address these challenges, we propose Reward-Decorrelated Policy Optimization (RDPO), a reward-processing method designed to explicitly target both failure modes. RDPO first utilizes Magnitude-Aware Quantile normalization to stabilize prompt-level advantage allocation across binary, fractional, and continuous rewards. It then applies Mahalanobis whitening within each active reward subspace to mitigate correlation redundancy prior to aggregation. When applied during the post-training of LongCat-Flash, RDPO enhances instruction following, writing quality, and robustness to hard prompts while remaining broadly competitive on reasoning and coding evaluations.
BibTeX
@misc{bai2026multi,
  author = {Yang Bai and Kaiyuan Liu and Ziyuan Zhuang and Jiahong Zhou and Rongxiang Weng and Xin Chen and Jingang Wang and Xunliang Cai},
  title = {{Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization}},
  year = {2026},
  eprint = {2605.13641},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.13641},
}
Figure