Action Dependent Baselines for Policy Gradient Variance Reduction

This direction is merged into another direction: policy gradient variance. Its last version stays readable and citable here.

Version 1 of 119 papers33 conceptsfrozen 2026-10-04

In reinforcement learning, the policy gradient theorem provides a way to improve an agent's behavior by following the gradient of expected rewards. However, these gradients are notoriously "noisy." A single trajectory might result in a high reward due to a lucky sequence of environment transitions or a specific choice of action, rather than the general quality of the policy. This high variance leads to slow learning and instability.

To combat this, researchers traditionally use a baseline—a function, typically the value function $V(s)$, that is subtracted from the observed return. This centers the learning signal, asking not "how much reward did I get?" but "how much better was this than average for this state?" While state-dependent baselines are standard, a growing body of research explores action-dependent baselines to further isolate and cancel out the noise inherent in the action selection process itself.

The Limits of State-Only Baselines

Standard policy gradient methods are often restricted to state-dependent baselines to ensure the gradient estimate remains unbiased. As Philip S. Thomas and Emma Brunskill note in Policy Gradient Methods for Reinforcement Learning with Function Approximation and Action-Dependent Baselines, this constraint prevents the use of descriptive action-dependent information that could more effectively center reward data.

The variance in a policy gradient can be decomposed into three main sources: the randomness of the starting state, the stochasticity of the environment's transitions, and the agent's own stochastic action selection. A state-only baseline $V(s)$ can only address the first two. It cannot account for the specific noise introduced when an agent chooses one action over another in the same state.

Breaking the Bias Barrier with Stein’s Identity

The primary reason action-dependent baselines were historically avoided is that simply subtracting a function of the action, $B(s, a)$, usually introduces mathematical bias into the gradient estimate. To solve this, researchers have turned to Stein’s Identity.

In Action-dependent Control Variates for Policy Optimization via Stein's Identity, Hao Liu and colleagues demonstrate how to construct a class of zero-expectation functions that serve as control variates. By leveraging the gradient of the policy itself, they can build complex, non-linear baselines that depend on the action but, by construction, integrate to zero. This allows the baseline to cancel out action-specific noise without shifting the mean of the gradient estimate. This approach is particularly effective in high-dimensional continuous control tasks, where the interaction between state and action is too complex for simple linear models to capture.

Exploiting Structure in High Dimensions

When an agent controls a robot with many joints or a multi-agent system, the action space is not just large—it is factorized. Cathy Wu and her collaborators explored this in Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines.

They realized that for a factorized policy (like a diagonal Gaussian), the variance of one action dimension can be reduced by conditioning its baseline on the other action dimensions. By evaluating how a specific joint's movement contributed to the reward relative to what all other joints were doing at that moment, the learner achieves much more precise credit assignment. Their work suggests that the benefits of action-dependent baselines actually scale up with the dimensionality of the problem.

Sampling the "Many-Actions" Alternative

Another perspective on reducing action-specific variance is to simply sample more. Michał Nauman and Marek Cygan propose the Many-Actions framework in On Many-Actions Policy Gradient.

Instead of taking one action and seeing one result, the agent uses a learned dynamics model to estimate the outcomes of multiple potential actions from the same state. By partitioning the variance into "marginalized policy variance" (from states) and "policy-dependent variance" (from actions), they provide a mathematical condition for when it is more efficient to sample more actions at a single state rather than extending the trajectory further into the future.

A Reality Check: The Mirage of Action Noise

Despite the theoretical appeal, there is a significant caveat. In The Mirage of Action-Dependent Baselines in Reinforcement Learning, George Tucker and his team argued that many reported gains from these baselines were actually due to implementation biases.

Their analysis revealed a crucial insight: in many standard RL benchmarks, the variance stemming from the action choice is actually quite small compared to the variance originating from the rest of the trajectory (the "future" rewards). If the dominant source of noise is what happens after the action is taken, even a perfect action-dependent baseline will only provide marginal improvements. This highlights that the effectiveness of these methods depends heavily on the specific "noise profile" of the environment being solved.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_action_dependent_baselines_for_policy_gr_70ca89cb,
  title = {Action Dependent Baselines for Policy Gradient Variance Reduction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/action-dependent-baselines-for-policy-gradient-variance-reduction/rdr_70ca89cb0ef94a17adcf7eb12e887b42},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_action_dependent_baselines_for_policy_gr_70ca89cb_v1,
  title = {Action Dependent Baselines for Policy Gradient Variance Reduction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/action-dependent-baselines-for-policy-gradient-variance-reduction/rdr_70ca89cb0ef94a17adcf7eb12e887b42/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}