Reward Decomposition and Multi Critic Architectures

133 concepts103 papers3 proposals

In standard reinforcement learning (RL), we typically compress everything we care about—speed, safety, energy efficiency, and behavioral style—into a single scalar reward. While mathematically convenient, this early scalarization creates a fundamental bottleneck. When we collapse a multi-dimensional objective space into a single number, we lose critical information about the environment's underlying dynamics, leading to unstable training, gradient conflicts, and uninterpretable policies.

To overcome this "scalarization trap," a growing body of research focuses on reward decomposition. Instead of mixing rewards into a single scalar during training, these approaches preserve rewards as a vector of distinct components and utilize decoupled multi-critic architectures to optimize them independently.

The Problem with the Scalar Trap

When multiple competing objectives are blended into a single scalar reward function, the learning process suffers from three core failure modes:

  1. Gradient Interference: Different objectives often push the policy in opposite directions. For instance, an autonomous vehicle trying to maximize speed while avoiding collisions receives conflicting signals. In a monolithic architecture, the gradients from these objectives can cancel each other out, resulting in a sub-optimal, confused policy.
  2. Signal Overshadowing (Scale Dominance): High-magnitude or highly frequent (dense) rewards completely drown out low-magnitude or infrequent (sparse) rewards. In humanoid robotics, for example, a large reward for forward velocity can easily overwhelm a small but critical penalty for losing balance, causing the robot to fall over in pursuit of speed.
  3. The Averaging Effect: In alignment and preference learning, collapsing distinct qualitative values—such as helpfulness and safety—into one score forces the model to learn an average of the two, failing to respect the unique boundaries of either constraint.

The Core Mechanism: Decoupled Value Estimation

The solution is to decompose the global reward signal into a vector of independent components:

$$\mathbf{R}(s, a) = [r_1(s, a), r_2(s, a), \dots, r_n(s, a)]$$

Instead of training a single critic network to estimate the expected cumulative return of a scalar sum, a multi-critic architecture assigns dedicated value function heads or entirely separate networks to evaluate each reward stream independently.

                  ┌───────────┐
                  │  Policy   │
                  │  (Actor)  │
                  └─────┬─────┘
                        │ Actions
                        ▼
         ┌──────────────┴──────────────┐
         │         Environment         │
         └─┬────────────┬────────────┬─┘
           │ r_1        │ r_2        │ r_n
           ▼            ▼            ▼
     ┌───────────┐┌───────────┐┌───────────┐
     │  Critic 1 ││  Critic 2 ││  Critic n │
     └───────────┘└───────────┘└───────────┘

By decoupling the value estimation, the single unified actor can query distinct critics to understand exactly why an action was beneficial or detrimental. As George Cheng demonstrates in Multi-Critic Actor Learning: Teaching RL Policies to Act with Style, keeping these value estimators separate ensures that conflicting reward signals from different objectives do not corrupt the gradient updates for the shared policy, allowing a single agent to master diverse behavioral styles without performance collapse.

Why It Works: Scale Isolation and Temporal Flexibility

Decoupling critics provides several mathematical and algorithmic advantages that stabilize policy optimization:

Independent Discounting and Normalization

Different objectives operate on different timescales and magnitudes. In recommendation systems, user clicks are frequent and dense, while long-term user retention is sparse and high-value. Qingpeng Cai et al. show in Two-Stage Constrained Actor-Critic for Short Video Recommendation that multi-critic reward decomposition allows designers to apply independent discount factors ($\gamma$) to each stream. This prevents dense, short-term signals from overshadowing sparse, long-term goals during value estimation.

Similarly, reward components can be normalized independently before being combined into an advantage signal. In The Overthinker's DIET: Cutting Token Calories with DIfficulty-AwarE Training, Weize Chen et al. utilize independent normalization for task-based rewards and efficiency penalties. This prevents high variance in task performance from inadvertently suppressing or wiping out the influence of the efficiency penalty during optimization.

Preventing Signal Overshadowing

In complex physical control tasks, separating the reward channels prevents primary objectives from drowning out critical sub-tasks. For instance, in Humanoid Whole-Body Locomotion on Narrow Terrain via Dynamic Balance and Reinforcement Learning, Weiji Xie leverages a multi-head critic network for reward vectorization. This architectural choice explicitly prevents dominant objectives like forward velocity from drowning out smaller-magnitude but vital signals like balance, significantly accelerating the learning process and yielding more robust control.

Mitigating Preference Averaging

In preference learning and AI alignment, users often have multi-faceted or conflicting criteria. In Preference Learning for AI Alignment: a Causal Perspective, Kasia Kobalczyk et al. argue that standard monolithic reward models suffer from learning spurious correlations. By using a shared representation layer followed by independent prediction heads for distinct objectives (such as safety and helpfulness), the architecture prevents the model from averaging out conflicting preferences, preserving the distinct boundaries of each qualitative goal.

Precision Optimization via Gradient Surgery

When rewards are decomposed, the policy optimization step can be managed with much higher precision. Instead of a single backpropagation pass that blends all objectives together, the policy can evaluate the directional alignment of the gradients from each critic.

For example, in bio-molecular design, optimizing an antibody requires balancing attractive and repulsive physical forces. Ruizhe Chen et al. demonstrate in Antigen-Specific Antibody Design via Direct Energy-based Preference Optimization that decomposing these complex reward signals into distinct components allows for targeted gradient surgery. If the gradient for avoiding physical clashes conflicts with the gradient for target binding, the optimization algorithm can project one gradient onto the normal plane of the other. This prevents the model from settling into degenerate solutions—such as avoiding physical clashes by simply moving the binder entirely away from the target.

Similarly, in robotics, Boliang Cai et al. show that separating navigation and collision avoidance into distinct critic networks prevents gradient conflicts between competing objectives, allowing the policy to respect the unique geometric requirements of safety tasks independently from goal-seeking behaviors.

Explainability and Human Alignment

Beyond training stability, reward decomposition fundamentally changes how we interpret an agent's behavior. When an agent makes a sub-optimal or unexpected move, a scalar reward system offers no insight into why the choice was made.

By breaking the scalar signal into distinct, semantically meaningful components, the agent's internal trade-offs are laid bare. As highlighted by Conor F. Hayes et al. in A practical guide to multi-objective reinforcement learning and planning, vector-based objectives allow agents to explicitly communicate the magnitude of trade-offs between competing goals. This transparency provides actionable insights into whether an agent failed due to a lack of capability or because it intentionally prioritized one constraint over another, moving reinforcement learning closer to safe, verifiable deployment in human-centric environments.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reward_decomposition_0b8b8be6,
  title = {reward decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reward-decomposition/rdr_0b8b8be63f0e44c989ec5f84a06d6348},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_reward_decomposition_0b8b8be6_v1,
  title = {reward decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reward-decomposition/rdr_0b8b8be63f0e44c989ec5f84a06d6348/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake