TL;DR
Behavior Regularized Policy Optimization (BRPO) prevents offline RL agents from taking catastrophic out-of-distribution actions, but has historically been limited to asymmetric divergences (like forward or reverse KL). This paper shows why symmetric regularization (such as Jensen-Shannon or Jeffreys) has been out of reach—it inherently lacks an analytic closed-form optimal policy. By using a truncated Taylor series of Pearson-Vajda $\chi^n$ divergences, the authors derive Symmetric $f$-Actor-Critic (S$f$-AC), the first practical offline RL algorithm with symmetric behavior regularization that avoids the per-environment failures common in standard baselines.
What Problem Does This Paper Address?
In offline reinforcement learning, agents learn strictly from static datasets without environment interaction. To prevent the policy from choosing out-of-distribution (OOD) actions with erroneously high value estimates, BRPO penalizes the divergence between the learned policy and the data-generating behavior policy:
$$\max_{\pi} \mathbb{E}[Q(s, a) - \tau D_{\text{Reg}}(\pi \,\|\, \pi_D)]$$
Standard methods rely exclusively on asymmetric divergences (e.g., KL, $\alpha$-, or Tsallis divergences). While symmetric divergences (like Jensen-Shannon) have demonstrated superior stability in domains like Large Language Model alignment (RLHF), adopting them in offline RL poses two major roadblocks:
- No Analytic Maximizer: The regularized optimal policy $\pi^*$ cannot be expressed in closed-form elementary functions under symmetric divergence regularization.
- Optimization Inconsistency & Instability: Attempting symmetric divergence minimization as an actor loss ($D_{\text{Opt}}$) while using an asymmetric regularizer ($D_{\text{Reg}}$) induces policy mismatch error. Naive symmetric losses also cause numerical breakdown (e.g., division by zero) when policies have bounded or disjoint support.
Methods
The authors introduce Symmetric $f$-Actor-Critic (S$f$-AC) to make symmetric regularization computationally tractable and stable:
- Polynomial Truncation of the $\chi^n$ Series: Any $f$-divergence can be expanded into an infinite series of Pearson-Vajda $\chi^n$-divergences. By truncating this series at order $N$, solving for the stationary point of the Lagrangian yields an $N$-th degree polynomial. Invoking the Abel-Ruffini theorem, an analytic closed-form policy exists only when $N < 5$. For $N=2$, the target policy simplifies cleanly to a thresholded ReLU-style policy: $$\pi^*(a|s) \propto \pi_D(a|s) \left[ 1 + \frac{Q(s,a) - V(s)}{\tau} \right]_+$$
- Decomposed Conditional Symmetry: Rather than expanding the entire symmetric objective into unstable powers of policy ratios, the authors leverage the structure $f(t) = t \ln t + g(t)$, where $g(t)$ is the conditional symmetry function. The $t \ln t$ term corresponds to standard advantage-weighted regression, while $g(t)$ is Taylor-expanded around $t = 1$.
- Ratio Clipping as Series Convergence: Because the Taylor expansion centers at $t = 1$, the policy ratio $\pi^*/\pi_\theta$ is clipped to $[1 - \epsilon, 1 + \epsilon]$. This yields a theoretical error bound ($O(\epsilon^{N+1}/(N+1)!)$) and provides a rigorous mathematical motivation for PPO-style ratio clipping in offline policy improvement.
- Dual-Network Implementation: To evaluate $\pi^*$ for actions sampled from the parameterized policy $\pi_\theta$, an auxiliary network $\pi_\zeta$ is trained alongside the actor via standard advantage regression.
Datasets
The authors evaluated their framework on:
- Synthetic 1D Gaussian Mixture: A distribution-matching sanity check approximating a bimodal Gaussian mixture.
- D4RL MuJoCo Benchmarks: 9 continuous-control datasets across HalfCheetah, Hopper, and Walker2d with
medium,medium-replay, andmedium-expertsplits. - Complex D4RL Environments: AntMaze (
umaze-diverse), Franka Kitchen (kitchen-complete), and Adroit Manipulation (pen,door,hammer,relocateacrosshumanandclonedregimes).
No new benchmarks were introduced; standard public D4RL environments were used.
Key Insights and Ideas
- The Abel-Ruffini Barrier: Symmetric $f$-divergences produce transcendental stationarity conditions (often requiring non-elementary functions like the Lambert $W$). Truncating via Pearson-Vajda $\chi^n$ turns the problem algebraic, but general closed-form solutions are mathematically impossible beyond quartic ($N \ge 5$) polynomials.
- Fast Decay of JS Coefficients: In the Taylor expansion of the Jensen-Shannon conditional symmetry term, the series coefficients decay at the rate $\frac{(-1)^n}{n(n-1)2^{n-1}}$. This rapid decay stabilizes higher-order powers of policy ratios, making JS significantly less sensitive to higher-order truncation than Jeffreys divergence (whose coefficients decay at $1/n$).
- Regularizer Matching Matters: When $D_{\text{Reg}} \neq D_{\text{Opt}}$, the actor update introduces an unavoidable mismatch error $\epsilon_{\text{Mismatch}}$. Aligning both objectives under symmetric regularizers directly addresses this gap.
Key Results
- Eliminating Per-Environment Failures: On the 9 D4RL MuJoCo tasks, S$f$-AC (Jensen-Shannon) was the only algorithm to consistently rank in the top-3 across all 9 tasks, avoiding the severe performance drops suffered by baselines (e.g., AWAC collapsing on
medexptasks or IQL/SQL struggling on specific locomotion setups). - Robustness across Hyperparameters: S$f$-AC proved largely insensitive to the truncation order $N_{\text{loss}} \in \{2, \dots, 7\}$ for Jensen-Shannon and showed consistent scores across clipping bounds $\epsilon \in [0.2, 100]$.
- Hard Tasks: On difficult sparse-reward and high-dimensional environments (AntMaze, Franka Kitchen, and Adroit), S$f$-AC was competitive with or outperformed state-of-the-art baselines like XQL and IQL without task-specific divergence hand-tuning.
Limitations and Future Work
- Auxiliary Actor Overhead: S$f$-AC introduces an extra parametric network ($\pi_\zeta$) to evaluate target policy probabilities, increasing training runtime (e.g., $\approx 10$ hours for 1M steps on an A100 station vs. $\approx 4$ hours for IQL/SQL).
- Restricted Truncation in Practice: Although the derivation supports $N \in \{2, 3, 4\}$, the authors simplify the target policy formulation to $N=2$ to avoid cumbersome radical algebra, leaving higher-order target derivations unexploited in practical implementations.
- Extension Beyond Offline RL: The authors mention connections to RLHF and preference optimization, but empirical evaluation is currently limited to continuous control offline MDPs. Exploring S$f$-AC for LLM alignment remains an open direction.
Summary updated