When training reinforcement learning (RL) agents or aligning large language models, we rarely optimize for a single isolated metric. A recommendation system must balance video watch time with user likes; an autonomous vehicle must balance speed with safety; a language model must balance helpfulness with conciseness.
The most intuitive way to handle these multi-objective problems is scalarization: multiplying each reward signal by a preference weight and summing them up into a single scalar reward. However, this naive approach frequently triggers a phenomenon known as objective collapse, where the agent hyper-optimizes one metric while completely neglecting the others.
To understand why multi-objective learning fails in these scenarios, we have to look under the hood at how disparate reward scales distort gradients during optimization.
The Mechanics of Gradient Dominance
Why does simple scalarization fail? The root cause is that different objectives naturally operate on vastly different numerical scales and units.
As noted in A distributional view on multi-objective policy optimization, combining physical distance with energy consumption creates an immediate mathematical imbalance. If one reward component spans from 0 to 1000 while another spans from 0 to 1, the gradient updates will be completely dominated by the larger-scale objective.
As Xie et al. (2025) and Basaklar et al. (2023) demonstrate, this numerical overshadowing prevents the agent from learning behaviors associated with smaller-scale rewards, even if the user explicitly assigns a high preference weight to them. Huang (2021) notes that simple additive combinations of these signals lead to training failures because the relative importance of objectives is not properly calibrated. Without explicit scale alignment, the search for a diverse Pareto frontier collapses into a single biased region, failing to capture the full range of possible trade-offs (Parisi et al., 2016).
This issue is exacerbated when aggregating feedback from different automated reward models. As Jang et al. (2023) and Lee et al. (2025) observe, different reward models produce scores on incompatible scales or with varying distributions, making it difficult to aggregate their feedback into a single training signal without causing severe training instability.
The Hidden Trap of Pre-Scalar Normalization
A common countermeasure to magnitude disparity is to normalize the rewards before summing them. However, the order of operations matters deeply, and standard normalization techniques can inadvertently introduce a new failure mode.
Girgis et al. (2026) expose a critical mathematical flaw in group-based RL algorithms (such as Group Relative Policy Optimization, or GRPO). If multiple rewards and penalties are combined into a single scalar before the group normalization step, the normalization process implicitly reweights each component by its within-group variance:
$$\text{Normalized Advantage} = \frac{(R_{\text{reward}} + R_{\text{penalty}}) - \mu}{\sigma_{\text{combined}}}$$
Because the denominator uses the combined variance, components with high variance naturally hijack the optimization process, overriding the intended fixed weights or Lagrangian multipliers.
Similarly, Chen et al. (2025) observe that high variance in task success rates can completely suppress efficiency penalties during normalization, preventing models from learning conciseness. Bazzan et al. (2023) describe this as a "squashing effect," where compressing dynamic rewards into a fixed range obscures critical variance, causing agents to over-prioritize stable objectives over highly variable, performance-critical ones. Furthermore, if the environment's reward distribution shifts or is poorly calibrated, fixed thresholds or weighting schemes fail entirely to maintain the intended balance (Smith et al., 2022; Yuan et al., 2025).
Learning Asymmetry and the Path of Least Resistance
Gradient dominance isn't just driven by raw numerical scale; it is also driven by signal clarity and density. Optimizers are fundamentally opportunistic: they will exploit the easiest path to maximize the aggregate scalar score.
Jafari et al. (2024) and Ghari et al. (2025) demonstrate that when multiple goals are averaged, the model often over-performs on the easiest metric while neglecting harder constraints. This creates a severe learning asymmetry (Dsouza et al., 2026), where an agent prioritizes an easily learned objective over a noisy or complex one because the easy objective provides a much clearer training signal.
| Reward Type | Signal Density | Optimization Risk |
|---|---|---|
| Dense / Easy (e.g., WatchTime, Velocity) | High frequency, clear credit assignment | Dominates the gradient; encourages reward hacking |
| Sparse / Hard (e.g., Shares, Energy Efficiency) | Low frequency, noisy or delayed feedback | Suppressed by the optimizer; completely ignored |
For instance, in short-video recommendation, dense signals like watch time easily overwhelm sparse, high-value signals like shares or likes (Cai et al., 2023)). This imbalance encourages reward hacking, where the model produces nonsensical or low-quality outputs that maximize a specific dominant reward at the expense of overall coherence (de Langis et al., 2024).
Restoring Balance to Multi-Objective Gradients
To prevent objective collapse, modern multi-objective frameworks shift away from static scalarization toward scale-invariant and dynamically balanced architectures.
Independent Normalization (Scalarized Advantages)
To fix the variance distortion in group-based optimization, Girgis et al. (2026) propose normalizing each reward and cost signal independently before combining them. By computing "scalarized advantages," the normalization process preserves the exact mathematical importance of the specified optimization weights, preventing high-variance objectives from overriding constraints.
Dynamic Weighting via Gradient Magnitudes
Rather than relying on tedious manual tuning or prior knowledge of objective distributions (Zhang et al., 2021), algorithms can dynamically adjust weights during training based on active gradient magnitudes (de Langis et al., 2024). Re-scaling gradients to ensure that safety and reward objectives operate on similar magnitudes prevents one objective from dominating the policy update process (Peng et al., 2021).
Distribution Space and Win-Rate Calibration
Some approaches bypass reward space entirely. Abdolmaleki et al. (2020) optimize policies in distribution space using KL-divergence constraints, achieving natural scale-invariance across disparate units. Alternatively, Lee et al. (2025) calibrate raw, incompatible reward scores into uniform pairwise win-rates before applying Pareto-frontier sampling, ensuring that no single reward model can dominate the final policy.
Key Foundational Work
Multi-objective Reinforcement Learning through Continuous Pareto Manifold Approximation (2016) — contributed 1 concept elements to this direction.
Balancing Multiple Sources of Reward in Reinforcement Learning (2000) — contributed 1 concept elements to this direction.