Symmetric Behavior Regularized Policy Optimization

Lingwei Zhu, Haseeb Shah, Zheng Chen, Martha White

TL;DR

This paper introduces the first comprehensive framework for symmetric Behavior Regularized Policy Optimization (BRPO) in offline reinforcement learning. By leveraging a finite series approximation of Pearson-Vajda divergences, the authors overcome long-standing mathematical hurdles to deliver a closed-form optimal policy and a numerically stable optimization objective. The resulting algorithm, Symmetric $f$-Actor-Critic (S$f$-AC), consistently outperforms traditional asymmetric methods on standard benchmarks.

What Problem Does This Paper Address?

Offline reinforcement learning (RL) is highly valuable because it allows agents to learn from pre-collected, static datasets without risky real-world exploration. However, it suffers from a major vulnerability: distribution shift. If the learning policy selects an action outside the dataset's coverage, the value function (critic) can produce wildly inaccurate, overoptimistic estimates, causing the policy to fail.

Behavior Regularized Policy Optimization (BRPO) fixes this by penalizing the learned policy when it drifts too far from the dataset's behavior policy ($\pi_D$). Traditionally, researchers use asymmetric distances like the Kullback-Leibler (KL) divergence for this penalty because they yield clean, closed-form mathematical solutions.

However, asymmetric regularizers have a blind spot: they suffer from "one-sided bias," meaning they heavily penalize rare actions even if those actions yield massive rewards. They also behave poorly near action boundaries. While symmetric divergences (like Jensen-Shannon or Jeffreys) naturally mitigate these biases, they have been historically avoided in BRPO because they do not permit closed-form solutions and cause severe numerical instability (like underflow or exploding gradients) during optimization.

Methods

The authors bridge this gap by introducing a universal BRPO framework that utilizes an infinite series of Pearson-Vajda $\chi$ divergences to represent any $f$-divergence.

Their core methodological breakthroughs include:

  1. Truncated Series for Analytic Solutions: By truncating the infinite series to a finite number of terms ($2 \le N_{Reg} < 5$), they unlock a closed-form expression for the optimal regularized policy. This policy naturally filters out low-value actions, effectively controlling out-of-distribution risks.
  2. Stable Optimization Surrogate: To prevent the numerical instability typical of symmetric objectives, they decouple the symmetric divergence into a standard $t \ln t$ term and a Taylor-expanded conditional symmetry function $g(t)$. This formulation avoids flipping policy ratios, preventing mathematical explosions when policy probabilities approach zero.
  3. Symmetric $f$-Actor-Critic (S$f$-AC): They implement these principles into a practical actor-critic algorithm that maintains projection geometry consistency by using the same symmetric divergence family for both regularization and optimization.

Datasets

The authors evaluated their approach on the standard D4RL benchmark, which includes:

  • MuJoCo Locomotion: Standard continuous control tasks (HalfCheetah, Hopper, Walker2d) across various dataset qualities (Medium, Medium-Replay, Medium-Expert).
  • Adroit: Challenging, high-dimensional dexterous robotic hand manipulation tasks (door, pen, relocate, hammer) characterized by highly narrow data distributions.
  • Maze2D: Sparse-reward navigation tasks requiring trajectory stitching.

Key Insights and Ideas

  • Mitigating One-Sided Bias: Asymmetric KL regularizers are overly conservative; they penalize an action simply because its baseline behavior probability is small. Symmetric regularizers look at the probability ratios more holistically, allowing the agent to safely exploit rare but highly rewarding actions.
  • Better Boundary Behavior: In continuous control, actions are typically clipped to a specific range (e.g., $[-1, 1]$). Asymmetric KL minimization tends to spread probability mass widely, causing a significant portion of the policy to spill outside the allowed range. Symmetric optimization penalizes mismatches on both sides, keeping the policy tightly within valid boundaries and doubling the post-clipping reward in didactic tests.
Figure 1: Symmetric divergences improve over one-sided divergences. (a)&(b): symmetric regu- larization helps mitigate support gaps and bias. (c)&(d): symmetric optimization improves policy behavior near the action boundary under clipping. The percentage in (d) indicates the proportion of probability mass falling off a = −1.
  • The Danger of Mismatched Geometry: Prior works occasionally used symmetric objectives for policy fitting, but paired them with asymmetric KL targets. The authors prove mathematically that this mismatch distorts the projection geometry, introducing unnecessary policy-representation errors.

Key Results

S$f$-AC (specifically using the Jensen-Shannon variant) achieved top-tier performance across the board:

  • On the D4RL MuJoCo suite, S$f$-AC consistently matched or outperformed highly competitive baselines like IQL, XQL, and AWAC.
  • On the more difficult Adroit tasks, S$f$-AC demonstrated substantial gains. For instance, on the pen-expert task, S$f$-AC achieved a score of 173.56, significantly outperforming IQL (128.05) and CQL (-1.41).
  • Ablation studies proved that the algorithm is highly robust to the choice of $N_{Loss}$ (the number of terms in the series expansion), showing stable performance across values from 2 to 6.

Limitations and Future Work

The primary limitation of this approach is that the Pearson-Vajda series approximation relies on the policy ratio remaining close to its convergence radius of 1. To guarantee this in practice, the authors had to implement PPO-style clipping on the policy ratios.

For future work, the authors aim to design a formulation that bypasses the explicit computation of the policy ratio entirely, thereby eliminating the need for heuristic clipping mechanisms.

Summary updated

Abstract

Behavior Regularized Policy Optimization (BRPO) leverages asymmetric divergence regularization to mitigate distribution shift in offline reinforcement learning. This paper is the first to study the open question of symmetric BRPO. Using didactic examples, we show that symmetric regularization can outperform asymmetric regularization in addressing one-sided bias, near-boundary policy updates, and projection geometry consistency. However, symmetric divergences do not fit BRPO naturally: they do not permit a closed-form solution when used as regularizers, and can lead to numerical instability when used as optimization objectives. We first introduce a universal BRPO framework using an infinite series of Pearson-Vajda divergences to represent any $f$-divergence, which includes both symmetric and asymmetric divergences. We use a finite-series approximation to obtain the following results for symmetric BRPO: (1) a closed-form optimal policy expression; (2) a numerically stable optimization surrogate; and (3) a tight upper bound on the approximation quality. On the D4RL benchmark and in didactic examples, we show that the proposed method achieves consistently strong results and is robust to the number of terms in the approximation.
BibTeX
@misc{zhu2025symmetric,
  author = {Lingwei Zhu and Haseeb Shah and Zheng Chen and Martha White},
  title = {{Symmetric Behavior Regularized Policy Optimization}},
  year = {2025},
  eprint = {2508.04225},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.04225},
}
Figure