TL;DR
WAS-PO (Whitened Advantage Summation Policy Optimization) addresses a hidden bias in multi-reward reinforcement learning: when rewards are correlated, standard summation overcounts their shared variance, distorting user-defined weights. By applying ZCA whitening to decorrelate advantages before aggregation, WAS-PO achieves significantly higher training rewards in complex tool-calling tasks, though it reveals a surprising risk of "inverting" minor rewards when correlations are extreme.
What Problem Does This Paper Address?
Modern Large Language Models (LLMs) are rarely trained on a single objective. Instead, researchers use Reinforcement Learning from Human Feedback (RLHF) to balance multiple rewards simultaneously—such as factual accuracy, output formatting, and response length.
The standard approach (like in the popular GRPO or GDPO algorithms) is to normalize each reward and then take a weighted sum. However, these rewards are often highly correlated. For example, a "format" reward might only be granted if the "accuracy" reward is already positive. When rewards move together, their shared variation is effectively counted multiple times in the final sum. This means the model isn't actually following the weights the researcher intended; instead, it over-prioritizes the common components of correlated rewards.
Methods
The authors propose Whitened Advantage Summation Policy Optimization (WAS-PO). The core idea is to decorrelate the reward signals before they are summed, ensuring that the chosen weights $w$ apply to independent (orthogonal) components of the reward space.
The process follows these key steps:
- Batch-level Normalization: Unlike GDPO, which normalizes rewards within a group of completions for a single prompt, WAS-PO normalizes rewards across the entire training batch.
- Correlation Estimation: It maintains a running estimate of the reward correlation matrix $\Sigma$ using an exponential moving average (EMA).
- ZCA Whitening: It applies a Zero-phase Component Analysis (ZCA) transform ($\Sigma^{-1/2}$) to the normalized advantages. ZCA is chosen because it produces decorrelated components that remain as close as possible to the original individual rewards.
- Weighted Aggregation: The whitened advantages are then summed using the user-specified weights.
The full procedure is detailed in the algorithm below:
Datasets
The researchers evaluated WAS-PO using two primary settings:
- Berkeley Function Calling Leaderboard (BFCL) v3: A dataset for tool-calling tasks. They used a subset of single-turn prompts and optimized for up to four rewards: accuracy, format, length, and structure.
- GSM8K: A standard math reasoning dataset. This was used specifically to study "ZCA-Inversion" by creating two nearly identical (highly correlated) rewards by construction.
Key Insights and Ideas
- Weight Distortion is Quantifiable: The authors prove that standard weighted summation actually applies weights of $\Sigma^{1/2}w$ to the decorrelated components of the reward signal. In their experiments, this caused the effective weight vector to point away from the intended direction by as much as $20.7^\circ$.
- The "ZCA-Inversion" Trap: A critical finding is that if two rewards are extremely correlated and have unequal weights, whitening can actually flip the sign of the lighter-weighted reward. The optimization then treats that reward as something to minimize.
- Batch vs. Per-Prompt Normalization: The paper highlights that moving from per-prompt normalization (comparing a model's answers to its own other attempts) to batch-level normalization (comparing answers across different prompts) significantly impacts training dynamics, often providing a "rising tide" that lifts training rewards across the board.
Key Results
WAS-PO showed a clear advantage in maximizing the training objective in complex scenarios:
- Higher Training Reward: In tool-calling tasks with three or four rewards, WAS-PO consistently outperformed GDPO, achieving a 3.8% to 3.9% higher mean training AUC.
- Training vs. Generalization: Interestingly, these gains in training reward did not translate to better performance on held-out data. In fact, held-out accuracy was slightly lower (by about 0.03 absolute) compared to GDPO in the four-reward setting.
- Empirical Inversion: In the math setting with nearly identical rewards ($\rho \approx 0.98$), WAS-PO successfully drove up accuracy while intentionally driving down the "format" reward, exactly as their theoretical "sign reversal" condition predicted.
Limitations and Future Work
- The Generalization Gap: The most significant limitation is that WAS-PO appears to "over-optimize" the training rewards without improving held-out accuracy. The authors suggest this might be due to the method sharpening the policy faster than it can generalize.
- Sensitivity to Estimation: Whitening depends heavily on an accurate estimate of the correlation matrix. If the matrix becomes nearly singular (e.g., two rewards become perfectly correlated), the whitening transform can become unstable.
- Practical Guidance: The authors advise against whitening safety rewards unless their weights are strictly constrained to be non-negative, to avoid the risk of the model learning to be "less safe" to maximize other objectives more efficiently.
Future work remains to determine if these findings hold for larger models and whether whitening can be tuned to improve generalization rather than just training performance.
Summary updated