Relative Entropy Regularization for Policy Optimization

89 concepts77 papers1 proposal

In reinforcement learning, training an agent often feels like walking a tightrope. Standard policy updates are greedy: they look at the current estimate of what actions are good and shift the policy directly toward them. However, if the value estimates are noisy or inaccurate, a large update can cause the policy to collapse, destroying previously learned behaviors or causing wild oscillations.

The root of this instability lies in a mismatch between parameter space and behavior space. When optimizing a neural network policy, a tiny change in the network's weights can cause an unpredictably massive shift in the resulting action distribution. To solve this, researchers use relative entropy policy regularization. By enforcing a mathematical "speed limit" on how much an agent's behavior can change between iterations, this approach transforms unstable policy updates into smooth, monotonic improvements.

The Core Intuition: Behavioral Trust Regions

To understand why relative entropy regularization works, consider how we measure distance. In standard deep learning, we update parameters using gradient descent, implicitly assuming that a small Euclidean distance in weight space ($\Delta \theta$) corresponds to a small change in performance. In reinforcement learning, this assumption is false. Silencing a single neuron could completely change an agent's strategy from safe navigation to driving off a cliff.

Instead of bounding parameter changes, relative entropy regularization bounds changes in the probability distributions of the actions themselves. The standard tool for this is the Kullback-Leibler (KL) divergence.

Standard Optimization:  [Old Weights] ----(Euclidean Step)----> [New Weights]  --> Unpredictable Behavior Shift
Regularized Search:     [Old Policy]  ----(KL Divergence)-----> [New Policy]   --> Bounded Behavior Shift

By integrating a relative entropy penalty directly into the optimization objective, as demonstrated by Mohammad Gheshlaghi Azar et al., the algorithm ensures that the new policy remains close to the previous iteration's baseline. This mechanism smooths out the optimization landscape, making the agent robust to approximation errors and preventing the deterministic collapse common in unregularized methods.

This behavioral constraint can be enforced either as a hard constraint (defining a strict trust region) or a soft penalty (adding the divergence to the loss function). As Alex X. Lee et al. observed, soft regularized methods often exhibit greater stability across varying conditions—such as when learning from a suboptimal teacher—because they avoid the hyperparameter sensitivity of hard-constrained boundaries.

Mathematical Foundations: Mirror Descent and Inference

Why does penalizing relative entropy yield such clean optimization properties? The answer connects reinforcement learning to mathematical optimization and probabilistic inference.

The Mirror Descent View

When we update a policy by balancing reward maximization with a KL penalty to the old policy, we are performing Online Mirror Descent on a statistical manifold. Jing Dong et al. leveraged this framework to guarantee stable convergence in robust Markov Decision Processes (MDPs). Similarly, Albert Eaton Shaw et al. showed that this can be framed as a stochastic dual ascent combined with a KL proximity mapping, ensuring that the policy evolves smoothly while remaining grounded in the dual optimization framework's requirements.

The Inference View

Alternatively, we can frame policy search as a probabilistic inference problem. Instead of asking "what actions maximize reward?", we ask "what is the posterior distribution of actions given that the agent succeeds?". Algorithms like Maximum a Posteriori Policy Optimisation (MPO), introduced by Abbas Abdolmaleki et al., use an Expectation-Maximization (EM) framework. They apply distinct KL constraints: a mode-seeking constraint to pinpoint optimal actions, and a moment-matching constraint to prevent the policy from dropping its entropy too quickly. This perspective replaces the need for heuristic entropy injection, as the trust region naturally preserves exploration.

Expanding the Boundary: State Distributions and Safety

While bounding the policy's action distribution prevents abrupt behavioral shifts at a specific state, it does not fully account for the long-term consequences of those actions. A small change in a policy can compound over time, leading the agent into entirely unvisited, dangerous regions of the state space.

To address this, Hany Abdulsamad et al. introduced dual KL divergence constraints that simultaneously bound both the policy update and the resulting shift in the state distribution. Bounding the state distribution ensures that local linearizations or physics approximations remain valid during optimization.

[Policy Update Bounded] ----> Prevents erratic actions at state S
[State Distribution Bounded] -> Prevents the agent from drifting into unknown environments entirely

This dual-bounding concept is vital for safe reinforcement learning. In constrained environments, Zuxin Liu et al. demonstrated that trust-region regularization via relative entropy is essential to prevent safety performance collapse. Without it, policies overfit to local data batches and inadvertently violate global safety limits.

Beyond the KL Divergence

While the standard KL divergence is the most common metric for relative entropy regularization, it is not always the optimal choice. Because the KL divergence is asymmetric and unbounded, it can suffer from gradient explosion or numerical instability in highly non-linear or adversarial settings. Researchers have therefore explored the broader family of $f$-divergences to fine-tune optimization dynamics:

  • Total Variation (TV) Divergence: Zhengpeng Xie et al. utilized a lower bound based on TV divergence instead of KL divergence. This approach provides a more robust mathematical lower bound for performance improvements, stabilizing high-variance deep neural network architectures.
  • Jensen-Shannon (JS) Divergence: In adversarial settings where sensors might be compromised, Chen Lv showed that using the bounded JS divergence prevents gradient explosion, allowing agents to maintain consistent decision-making under observation uncertainty.
  • Hellinger Distance: Boris Belousov et al. analyzed generalized entropic regularizers, demonstrating that intermediate divergences like the Hellinger distance can outperform standard KL approaches by balancing the rate of policy change with the robustness of value estimates.

Structural and Architectural Extensions

The principle of bounding behavioral drift scales across various complex architectures, from robotics to large language models.

Multi-Agent and Hierarchical Systems

In multi-agent environments, an agent's environment is non-stationary because other agents are learning simultaneously. Zongqing Lu et al. addressed this by incorporating both a standard KL and a square-root KL term to bound the influence of concurrent policy shifts from other agents. For cooperative games, Yulai Zhao et al. proved that sequential updates regularized by KL divergence within a mirror descent framework guarantee global convergence.

In hierarchical reinforcement learning, Christian Daniel et al. applied relative entropy constraints to keep options (sub-policies) distinct, preventing them from collapsing into identical behaviors.

Decoupled Exploration and Alignment

Relative entropy constraints are also highly effective for managing internal model components:

  • Exploration vs. Exploitation: Lukas Schäfer used KL constraints to limit the behavioral gap between separate exploration and exploitation policies, preventing the exploration agent from gathering irrelevant data.
  • LLM Preference Alignment: When aligning large language models with human preferences, Yuheng Zhang et al. implemented a dual regularization strategy that constrains the model against both its original reference capabilities and its preceding training iteration, preventing policy oscillation. To make this process more flexible, Gang Li et al. introduced a squared-hinge penalty on the KL divergence, which activates only when the model's divergence exceeds a specific threshold.

Whether applied to continuous robotic control (Alexandros Paraschos et al.) or statistical manifolds in molecular geometry (Miloslav Torda et al.), relative entropy regularization provides a fundamental truth: by constraining optimization to the natural geometry of probability distributions, we ensure that learning remains stable, safe, and continuous.

Key Foundational Work

Dynamic Policy Programming with Function Approximation (2011) — contributed 2 concept elements to this direction.
Relative Entropy Policy Search (2010) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_relative_entropy_policy_regularization_a410a57e,
  title = {relative entropy policy regularization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-entropy-policy-regularization/rdr_a410a57ebb1e4179b1450e900d45aaf5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_relative_entropy_policy_regularization_a410a57e_v1,
  title = {relative entropy policy regularization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-entropy-policy-regularization/rdr_a410a57ebb1e4179b1450e900d45aaf5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake