Whitened Advantage Summation Policy Optimization: Correlation-Aware Aggregation for Multi-Reward Reinforcement Learning

Olaitan Olaleye, Jayson Meribe, Temidayo OlaleyearXiv 2026 · arXiv

TL;DR

WAS-PO (Whitened Advantage Summation Policy Optimization) addresses a hidden bias in multi-reward reinforcement learning: when rewards are correlated, standard summation overcounts their shared variance, distorting user-defined weights. By applying ZCA whitening to decorrelate advantages before aggregation, WAS-PO achieves significantly higher training rewards in complex tool-calling tasks, though it reveals a surprising risk of "inverting" minor rewards when correlations are extreme.

What Problem Does This Paper Address?

Modern Large Language Models (LLMs) are rarely trained on a single objective. Instead, researchers use Reinforcement Learning from Human Feedback (RLHF) to balance multiple rewards simultaneously—such as factual accuracy, output formatting, and response length.

The standard approach (like in the popular GRPO or GDPO algorithms) is to normalize each reward and then take a weighted sum. However, these rewards are often highly correlated. For example, a "format" reward might only be granted if the "accuracy" reward is already positive. When rewards move together, their shared variation is effectively counted multiple times in the final sum. This means the model isn't actually following the weights the researcher intended; instead, it over-prioritizes the common components of correlated rewards.

Methods

The authors propose Whitened Advantage Summation Policy Optimization (WAS-PO). The core idea is to decorrelate the reward signals before they are summed, ensuring that the chosen weights $w$ apply to independent (orthogonal) components of the reward space.

The process follows these key steps:

  1. Batch-level Normalization: Unlike GDPO, which normalizes rewards within a group of completions for a single prompt, WAS-PO normalizes rewards across the entire training batch.
  2. Correlation Estimation: It maintains a running estimate of the reward correlation matrix $\Sigma$ using an exponential moving average (EMA).
  3. ZCA Whitening: It applies a Zero-phase Component Analysis (ZCA) transform ($\Sigma^{-1/2}$) to the normalized advantages. ZCA is chosen because it produces decorrelated components that remain as close as possible to the original individual rewards.
  4. Weighted Aggregation: The whitened advantages are then summed using the user-specified weights.

The full procedure is detailed in the algorithm below:

Algorithm 1: The WAS-PO advantage computation process.

Datasets

The researchers evaluated WAS-PO using two primary settings:

  • Berkeley Function Calling Leaderboard (BFCL) v3: A dataset for tool-calling tasks. They used a subset of single-turn prompts and optimized for up to four rewards: accuracy, format, length, and structure.
  • GSM8K: A standard math reasoning dataset. This was used specifically to study "ZCA-Inversion" by creating two nearly identical (highly correlated) rewards by construction.

Key Insights and Ideas

  • Weight Distortion is Quantifiable: The authors prove that standard weighted summation actually applies weights of $\Sigma^{1/2}w$ to the decorrelated components of the reward signal. In their experiments, this caused the effective weight vector to point away from the intended direction by as much as $20.7^\circ$.
  • The "ZCA-Inversion" Trap: A critical finding is that if two rewards are extremely correlated and have unequal weights, whitening can actually flip the sign of the lighter-weighted reward. The optimization then treats that reward as something to minimize.
  • Batch vs. Per-Prompt Normalization: The paper highlights that moving from per-prompt normalization (comparing a model's answers to its own other attempts) to batch-level normalization (comparing answers across different prompts) significantly impacts training dynamics, often providing a "rising tide" that lifts training rewards across the board.

Key Results

WAS-PO showed a clear advantage in maximizing the training objective in complex scenarios:

  • Higher Training Reward: In tool-calling tasks with three or four rewards, WAS-PO consistently outperformed GDPO, achieving a 3.8% to 3.9% higher mean training AUC.
  • Training vs. Generalization: Interestingly, these gains in training reward did not translate to better performance on held-out data. In fact, held-out accuracy was slightly lower (by about 0.03 absolute) compared to GDPO in the four-reward setting.
  • Empirical Inversion: In the math setting with nearly identical rewards ($\rho \approx 0.98$), WAS-PO successfully drove up accuracy while intentionally driving down the "format" reward, exactly as their theoretical "sign reversal" condition predicted.
Figure 1: Training reward of GRPO, GDPO and WAS-PO. Curves and ±1 SD bands are computed over a subset of the seeds (three at K = 2). The per-seed AUCs in Tables 7 and 5 cover all seven seeds.

Limitations and Future Work

  • The Generalization Gap: The most significant limitation is that WAS-PO appears to "over-optimize" the training rewards without improving held-out accuracy. The authors suggest this might be due to the method sharpening the policy faster than it can generalize.
  • Sensitivity to Estimation: Whitening depends heavily on an accurate estimate of the correlation matrix. If the matrix becomes nearly singular (e.g., two rewards become perfectly correlated), the whitening transform can become unstable.
  • Practical Guidance: The authors advise against whitening safety rewards unless their weights are strictly constrained to be non-negative, to avoid the risk of the model learning to be "less safe" to maximize other objectives more efficiently.

Future work remains to determine if these findings hold for larger models and whether whitening can be tuned to improve generalization rather than just training performance.

Summary updated

Abstract

Reinforcement learning for language models often combines several rewards by normalizing each one and adding them with fixed weights. When the rewards are correlated, the variation they share is counted once for every reward that carries it. In terms of the ZCA-decorrelated components of the reward signal, the weighted sum applies weights $Σ^{1/2} w$ rather than the chosen weights $w$, where $Σ$ is the reward correlation matrix. We study WAS-PO, which whitens the normalized advantages with $Σ^{-1/2}$ before adding them, so that the chosen weights apply to those components. We derive the scalarized objective whose gradient it follows when normalization statistics are held fixed, show that its weights on the raw rewards can be negative, give an exact two-reward condition under which the lighter-weighted reward is reversed, and give a loose bound on the error from estimating $Σ$ during training. On tool calling with Qwen2.5-1.5B-Instruct, WAS-PO reached a higher mean training reward than GDPO in all seven seeds with three rewards (3.8% higher) and all three seeds with four (3.9% higher). WAS-PO also replaces GDPO's per-prompt normalization with batch-level normalization, and we did not separate the two changes with three or four rewards. The training-reward gains did not carry over to held-out accuracy, which was slightly lower than GDPO's with three rewards (not significant) and 0.03 lower in absolute terms with four (two seeds evaluated). With two rewards, WAS-PO trailed GDPO in training reward, and a three-seed control weakly suggests that the gap comes from batch-level normalization rather than whitening. In a math setting where the two rewards were nearly identical by construction, WAS-PO drove down the lighter-weighted reward, as the sign condition predicts.
BibTeX
@misc{olaleye2026whitened,
  author = {Olaitan Olaleye and Jayson Meribe and Temidayo Olaleye},
  title = {{Whitened Advantage Summation Policy Optimization: Correlation-Aware Aggregation for Multi-Reward Reinforcement Learning}},
  year = {2026},
  eprint = {2611.02609},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02609},
}
Figure