Symmetric Behavior Regularized Policy Optimization

Lingwei Zhu, Haseeb Shah, Zheng Chen, Yukie Nagai, Martha White

Abstract: Behavior Regularized Policy Optimization (BRPO) leverages asymmetric (divergence) regularization to mitigate the distribution shift in offline Reinforcement Learning. This paper is the first to study the open question of symmetric regularization. We show that symmetric regularization does not permit an analytic optimal policy $\pi^*$, posing a challenge to practical utility of symmetric BRPO. We approximate $\pi^*$ by the Taylor series of Pearson-Vajda $\chi^n$ divergences and show that an analytic policy expression exists only when the series is capped at $n=5$. To compute the solution in a numerically stable manner, we propose to Taylor expand the conditional symmetry term of the symmetric divergence loss, leading to a novel algorithm: Symmetric $f$-Actor Critic (S$f$-AC). S$f$-AC achieves consistently strong results across various D4RL MuJoCo tasks. Additionally, S$f$-AC avoids per-environment failures observed in IQL, SQL, XQL and AWAC, opening up possibilities for more diverse and effective regularization choices for offline RL.

Introduction

In offline Reinforcement Learning (RL), agents learn from fixed datasets without the ability to interact with the environment. A primary hurdle is "distributional shift": the agent might choose an action not present in the dataset, receive a spuriously high predicted value for it, and fail during deployment.

To mitigate this, researchers use Behavior Regularized Policy Optimization (BRPO), which penalizes the agent for straying from the "behavior policy" (the actions seen in the data). Traditionally, this is done using asymmetric divergences like the Kullback-Leibler (KL) divergence. This paper introduces Symmetric $f$-Actor Critic (S$f$-AC), addressing the long-standing challenge of using symmetric divergences (like Jensen-Shannon or Jeffreys) which typically lack a closed-form analytic expression for the optimal policy $\pi^*$.

Overcoming the "Analytic Wall"

The authors tackle the lack of an analytic solution through a combination of algebraic theory and numerical approximation:

  1. The $n=5$ Wall: They prove that while the optimal policy can be approximated using a Taylor series of Pearson-Vajda $\chi^n$ divergences, an analytic solution only exists if the series is truncated at $n < 5$. This connects to the Abel-Ruffini theorem, which states there is no general algebraic solution for polynomial equations of degree five or higher.
  2. Symmetric $f$-Actor Critic (S$f$-AC): To ensure numerical stability, they decompose the symmetric divergence into a standard KL term and a "conditional symmetry" term.
  3. Taylor Expansion & Clipping: By Taylor-expanding only the conditional symmetry term and borrowing clipping techniques from PPO, they maintain stable updates within the neighborhood of the behavior policy.
Table 1: Summary of asymmetric and symmetric divergences and their corresponding generator functions f(t).

Empirical Evaluation

The authors evaluated S$f$-AC on the D4RL MuJoCo suite, including tasks like HalfCheetah, Hopper, and Walker2d across Medium, Medium-Replay, and Medium-Expert levels. They also tested on high-dimensional robotic manipulation (Adroit), multi-step cooking (Franka Kitchen), and long-term planning (AntMaze).

Key Insights

  • Optimization Alignment: The divergence used for the theoretical objective should match the one used for practical optimization to avoid "optimization mismatch error."
  • Robustness: Unlike popular algorithms (e.g., IQL or AWAC) that may fail on specific tasks, symmetric regularization appears significantly more stable across diverse environments.
  • Geometric Alignment: Symmetric divergences provide a balanced regularization, keeping the policy better centered within the data distribution compared to asymmetric KL.

Results

S$f$-AC (specifically the Jensen-Shannon variant) demonstrated superior consistency, being the only algorithm to rank in the top 3 across all 9 D4RL MuJoCo tasks. It frequently outperformed state-of-the-art baselines like IQL, SQL, and XQL in head-to-head comparisons.

Figure 1: Consistency analysis showing the number of MuJoCo tasks where each algorithm ranks in the top 3. S$f$-AC variants show significantly higher reliability than baselines.

Conclusion

S$f$-AC provides a mathematically grounded and numerically stable way to leverage symmetric regularization in offline RL. While the Taylor expansion introduces an approximation, the empirical results suggest that the benefits of symmetric alignment outweigh the costs of truncation. Future work may focus on refining hyperparameter sensitivity and exploring non-Taylor-based analytic approximations.

Summary updated

Abstract

Behavior Regularized Policy Optimization (BRPO) leverages asymmetric (divergence) regularization to mitigate the distribution shift in offline Reinforcement Learning. This paper is the first to study the open question of symmetric regularization. We show that symmetric regularization does not permit an analytic optimal policy $π^*$, posing a challenge to practical utility of symmetric BRPO. We approximate $π^*$ by the Taylor series of Pearson-Vajda $χ^n$ divergences and show that an analytic policy expression exists only when the series is capped at $n=5$. To compute the solution in a numerically stable manner, we propose to Taylor expand the conditional symmetry term of the symmetric divergence loss, leading to a novel algorithm: Symmetric $f$-Actor Critic (S$f$-AC). S$f$-AC achieves consistently strong results across various D4RL MuJoCo tasks. Additionally, S$f$-AC avoids per-environment failures observed in IQL, SQL, XQL and AWAC, opening up possibilities for more diverse and effective regularization choices for offline RL.
BibTeX
@misc{zhu2025symmetric,
  author = {Lingwei Zhu and Haseeb Shah and Zheng Chen and Yukie Nagai and Martha White},
  title = {{Symmetric Behavior Regularized Policy Optimization}},
  year = {2025},
  eprint = {2508.04225},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.04225},
}
Figure