TL;DR
Training Large Language Models (LLMs) using Reinforcement Learning (RL) often requires balancing multiple, conflicting reward signals (like correctness, formatting, and length), which frequently causes training instability. This paper introduces Reward-Decorrelated Policy Optimization (RDPO), a lightweight pipeline that uses quantile mapping to normalize diverse reward scales and Mahalanobis whitening to eliminate redundant correlations. When applied to the post-training of the LongCat-Flash model, RDPO significantly boosts instruction-following and writing quality while maintaining top-tier reasoning and coding performance.
What Problem Does This Paper Address?
In modern LLM alignment, a single training run usually juggles multiple objectives simultaneously. For example, a model needs to generate text that is factually accurate, adheres to complex formatting constraints, follows stylistic rubrics, and remains concise.
When we feed these diverse signals into standard RL frameworks like GRPO, we hit two major roadblocks:
- Scale and Distribution Mismatch: A binary pass/fail reward (e.g., code execution) behaves entirely differently from a continuous preference score or a quadratic length penalty. Standard linear normalization (Z-scoring) is highly sensitive to outliers and skewed distributions, often causing a single anomalous rollout to hijack the entire gradient update for a prompt.
- Inter-Reward Correlation: Many reward dimensions are inherently correlated. For instance, longer responses might naturally score higher on a general preference model, or thorough mathematical reasoning might require more tokens. Simply summing these rewards double-counts redundant information or inadvertently cancels out useful variance.
Methods
RDPO mitigates these issues through a lightweight, two-step reward processing pipeline executed before rewards are aggregated into a scalar advantage:
1. Magnitude-Aware Quantile (MAQ) Normalization
Instead of relying on standard Z-scores, RDPO maps rewards to a standard normal distribution based on their relative ranks within a prompt's rollout group. Crucially, to avoid losing the actual physical differences between scores (which happens in pure rank transformation), MAQ introduces log-compressed magnitude gaps. This preserves local quantitative nuances while compressing extreme outliers, ensuring stable advantage allocation across all rollouts.
2. Subspace Mahalanobis Whitening
To eliminate cross-reward redundancy, RDPO maintains a running estimate of the reward covariance matrix using an Exponential Moving Average (EMA). It then applies Mahalanobis whitening to decorrelate the dimensions, effectively transforming them into orthogonal, unit-variance signals.
Because different tasks activate different subsets of rewards (e.g., math tasks don't use formatting rubrics), RDPO applies this whitening dynamically only within the active reward subspace of the given task. This prevents the model from calculating artificial correlations between rewards that never co-occur.
Datasets
The authors validated RDPO during the post-training phase of the LongCat-Flash model using a rich mixture of four task categories: mathematical reasoning, code generation, instruction following, and general writing.
To evaluate the model, they leveraged a comprehensive suite of established benchmarks:
- Instruction Following: IFEval, GuideBench, SOP-Maze
- Math & Knowledge Reasoning: AIME24, AIME25, GPQA, MATH500
- Writing & Arena Evaluation: WritingBench, ArenaHard-v2 (including AH-Hard and AH-Creative subsets)
- Coding: FullStackBench, HumanEval+, MBPP+, LiveCodeBench v6
Key Insights and Ideas
- The Danger of Advantage Domination: The authors show that under standard Z-score normalization (GDPO), the advantage mass frequently concentrates on a single rollout. MAQ successfully counteracts this, lowering advantage domination and boosting effective rollout participation across all task types.
- Measuring "Information Efficiency": The paper introduces a novel diagnostic metric, Effective Information Efficiency ($\eta_{eff}$), which quantifies whether an aggregation method balances weights evenly and avoids redundant variance. RDPO maintains a substantially higher and more stable information efficiency throughout the entire training trajectory compared to baselines.
- Subspace-Specific Processing is Essential: In multi-task learning, a global whitening matrix fails because reward dimensions don't always overlap. Dynamically isolating the active reward subspace ensures that decorrelation only happens where it mathematically makes sense.
Key Results
The authors first performed a small-scale ablation study on a smaller model family, demonstrating that combining MAQ and Whitening (Q+M) yields a compounding benefit (e.g., IFEval scores jumped from a baseline of 78.56 to 83.55).
When scaled up to the full LongCat-Flash model, RDPO achieved standout results:
- Instruction Following & Writing: RDPO achieved 90.39% on IFEval (vs. GRPO's 89.46%) and 87.63% on WritingBench (vs. GRPO's 84.12%).
- Hard Prompts: On the challenging ArenaHard-v2 (Hard) benchmark, RDPO surged to 76.10%, a massive leap over GRPO (65.80%) and the initialization baseline (49.40%).
- Reasoning & Coding Consistency: While optimizing heavily for alignment and style constraints usually degrades core capabilities, RDPO remained highly competitive, matching or slightly exceeding GRPO on MATH500 (98.60%) and LiveCodeBench (63.80%).
Limitations and Future Work
- Covariance Warmup Dependency: The Mahalanobis whitening step relies on an accurate running covariance matrix. At the very beginning of training, this estimate is unstable, requiring a manual "warmup phase" (5 steps in this work) before whitening can be safely applied.
- Heuristic Gating for Core Constraints: To prevent auxiliary rewards (like length penalties) from overriding core task success, the authors still had to implement manual conditional rules (e.g., truncating preference rewards if a core rubric fails). Natively incorporating these hierarchical constraints directly into the decorrelation mathematics remains an open area for future research.
Summary updated