Policy Gradients via Analytical Integration over Action Spaces

This direction is merged into another direction: policy gradient variance. Its last version stays readable and citable here.

Version 1 of 113 papers25 conceptsfrozen 2026-10-04

In standard reinforcement learning, an agent learns by trial and error: it samples an action, observes a reward, and adjusts its policy to make high-reward actions more likely. This sampling process is the primary source of variance in policy gradient methods. If an agent happens to pick a "lucky" action in a bad state, the resulting gradient update can be misleadingly large, leading to unstable training.

A growing body of research suggests a more stable alternative: instead of sampling a single action to compute a gradient, we should analytically integrate over the entire action distribution. By evaluating the "mean" behavior of the policy rather than a single realization, we can drastically reduce noise and unify disparate RL paradigms.

The Problem with Sampling

Most policy gradient methods rely on the likelihood ratio (or REINFORCE) estimator. This estimator asks: "If I took action $a$ and it was good, how should I change my policy parameters to make $a$ more probable?" Because this depends on a single stochastic sample, the gradient signal is often "noisy."

Kavosh Asadi and George Konidaris addressed this in Mean Actor Critic by proposing that in discrete action spaces, we don't need to sample at all during the policy update. Instead of using the value of the sampled action, they calculate the expected value by summing over all possible actions: $\sum_a \pi(a|s) Q(s, a)$. This "Mean Actor-Critic" (MAC) approach eliminates the randomness inherent in action selection, providing a gradient that is theoretically guaranteed to have lower variance than traditional sampled methods.

Treating Distributions as Actions

While summing over actions works for discrete sets, continuous action spaces (like joint torques in robotics) require a different approach. One powerful intuition is to shift the perspective of what an "action" actually is.

In Distributions as Actions: A Unified Framework for Diverse Action Spaces, Jiamin He, A. Rupam Mahmood, and Martha White argue that the agent's action should be viewed as the parameters of the distribution (e.g., the mean and variance of a Gaussian) rather than a sample from it. By redefining the problem this way, the policy gradient can be calculated analytically in the parameter space. This "DA-AC" framework turns discrete, continuous, and hybrid spaces into a single continuous control problem, leveraging analytical integration to maintain a low-variance signal.

This echoes earlier work by Jürgen Schmidhuber and Daan Wierstra in Policy Gradient Critics. They introduced a critic that evaluates probability distributions over actions rather than discrete points. By using LSTMs to handle memory and evaluating the distribution itself, their PGAC algorithm allows agents to learn the optimal level of stochasticity (exploration) explicitly, rather than relying on a hard-coded noise schedule.

The Stochastic-Deterministic Continuum

This shift toward analytical evaluation also helps bridge the gap between two major schools of RL: Deterministic Policy Gradients (DPG) and Maximum Entropy RL (like Soft Actor-Critic).

Tuomas Haarnoja and Sergey Levine demonstrated in Reinforcement Learning with Deep Energy-Based Policies that deterministic updates are actually a special case of a broader entropy-regularized framework. When you remove the entropy-driven sampling term from a "soft" update, you recover the standard deterministic update rule.

This unification is further explored by Philip Ball in OffCon3: What is state of the art anyway?. Ball argues that algorithms like SAC and TD3 are not rivals but are points on a continuous spectrum of "Off-Policy Continuous Generalized Policy Iteration." In this view, stochastic value gradients serve as the bridge: in deterministic environments, the variance of a stochastic policy naturally collapses toward zero, transforming the stochastic update into a deterministic one.

Why It Works: Smoothing the Landscape

The core reason these methods succeed is smoothing. A critic that evaluates a single action $Q(s, a)$ sees a jagged landscape. A critic that evaluates a distribution—effectively $\int \pi(a|s) Q(s, a) da$—sees a smoothed version of that landscape.

By integrating over the action space, the agent:

  1. Reduces Variance: It ignores the "luck" of a single sample and focuses on the average utility of its current strategy.
  2. Unifies Action Spaces: It treats discrete choices and continuous ranges as the same mathematical object (a distribution).
  3. Stabilizes Convergence: The analytical gradient provides a more consistent direction for optimization, preventing the "oscillations" common in high-noise reinforcement learning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_policy_gradients_via_analytical_integrat_3ecdb4e8,
  title = {Policy Gradients via Analytical Integration over Action Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/policy-gradients-via-analytical-integration-over-action-spaces/rdr_3ecdb4e8024542938a3c8efac38426eb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_policy_gradients_via_analytical_integrat_3ecdb4e8_v1,
  title = {Policy Gradients via Analytical Integration over Action Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/policy-gradients-via-analytical-integration-over-action-spaces/rdr_3ecdb4e8024542938a3c8efac38426eb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}