Policy Gradient Bayesian Robust Optimization for Imitation Learning

Zaynah Javed, Daniel S. Brown, Satvik Sharma, Jerry Zhu, Ashwin Balakrishna, Marek Petrik, Anca D. Dragan, Ken Goldberg

Abstract

The difficulty in specifying rewards for many real-world problems has led to an increased focus on learning rewards from human feedback, such as demonstrations. However, there are often many different reward functions that explain the human feedback, leaving agents with uncertainty over what the true reward function is. While most policy optimization approaches handle this uncertainty by optimizing for expected performance, many applications demand risk-averse behavior. We derive a novel policy gradient-style robust optimization approach, PG-BROIL, that optimizes a soft-robust objective that balances expected performance and risk. To the best of our knowledge, PG-BROIL is the first policy optimization algorithm robust to a distribution of reward hypotheses which can scale to continuous MDPs. Results suggest that PG-BROIL can produce a family of behaviors ranging from risk-neutral to risk-averse and outperforms state-of-the-art imitation learning algorithms when learning from ambiguous demonstrations by hedging against uncertainty, rather than seeking to uniquely identify the demonstrator's reward function.

Introduction

When learning from human demonstrations, AI agents often face ambiguity about the true reward function. This epistemic uncertainty—uncertainty about the objective itself—can lead to unsafe or misaligned behavior if an agent simply optimizes for the most likely reward hypothesis. Standard Reinforcement Learning (RL) and Imitation Learning (IL) methods typically pick a single reward estimate (like the mean or MAP) and optimize it blindly, risking "reward hacking" or catastrophic failure if the estimate is slightly off.

While prior work has explored robust optimization to handle this, existing methods are often overly pessimistic (assuming a worst-case adversary) or limited to simple, discrete environments with known dynamics. PG-BROIL addresses these challenges by bringing risk-averse, Bayesian reward uncertainty reasoning into complex, continuous environments with unknown dynamics.

Methodology: Soft-Robust Optimization

PG-BROIL builds on the Bayesian Robust Optimization for Imitation Learning (BROIL) framework, which optimizes a "soft-robust" objective. Instead of maximizing only the expected return, it maximizes a weighted sum of the Expected Return and the Conditional Value at Risk (CVaR) across a distribution of possible reward functions.

Visual representation of Value at Risk (VaR) and Conditional Value at Risk (CVaR) on a reward distribution.

The core technical contribution is the derivation of a Policy Gradient for BROIL (PG-BROIL). CVaR is a powerful metric for measuring tail risk (the worst-case scenarios), but its non-differentiability makes it difficult to integrate with standard deep RL policy gradient methods. The authors overcome this by proving that an approximate subgradient for the CVaR objective can be computed efficiently. By evaluating trajectory rollouts across a finite set of sampled reward hypotheses and solving a quick line search to find the risk threshold, they calculate a weighted policy gradient. This allows PG-BROIL to be integrated into scalable algorithms like PPO or TRPO.

Experimental Evaluation

The authors evaluate PG-BROIL on several continuous control environments modified to include regions of uncertain cost:

  • CartPole: Reward is an unknown linear function of the cart's position.
  • Pointmass Navigation: A 2D task with "gray regions" of uncertain cost (e.g., potentially damaging terrain).
  • Reacher: A robotic arm must reach a goal while navigating around a red zone of uncertain penalty.
  • TrashBot: A custom domain where a robot picks up trash while avoiding uncertain boundary regions.
  • Atari Boxing: Learned from suboptimal demonstrations to balance scoring hits with avoiding damage.

Key Insights and Results

A primary insight of this work is that hedging beats guessing. In imitation learning, uniquely identifying a demonstrator's exact reward function from limited data is often impossible. It is safer to maintain a posterior distribution of likely reward functions and optimize a policy that performs well under all of them.

PG-BROIL provides tunable risk sensitivity via a parameter $\lambda$. This acts as a dial between risk-neutrality ($\lambda=1$, maximizing expected reward) and high risk-aversion ($\lambda=0$, protecting against worst-case plausible rewards).

Trajectories in the Pointmass Navigation task for different values of the risk-aversion parameter λ.

Experimental results show that PG-BROIL consistently outperforms state-of-the-art baselines:

  • TrashBot: PG-BROIL collected 8.4 pieces of trash while spending nearly zero steps (0.1) in dangerous regions, whereas baselines like Behavioral Cloning (BC) frequently entered unsafe zones.
  • Reacher with Domain Shift: PG-BROIL successfully avoided uncertain regions (1.7 steps inside) in new layouts, while Bayesian REX (13.5 steps) confidently entered them.
  • Atari Boxing: PG-BROIL achieved a score of 23.9, vastly outperforming BC (1.7) and GAIL (-0.2) by learning to balance offensive and defensive play.

Limitations and Future Work

The algorithm can become unstable during training at extreme risk-aversion levels ($\lambda \approx 0$) due to the hard indicator function used in the CVaR policy gradient. Future work may address this by exploring continuously differentiable risk measures, such as the Entropic Risk Measure, or scaling the approach to raw pixel inputs using contrastive learning.

Summary updated

Abstract

The difficulty in specifying rewards for many real-world problems has led to an increased focus on learning rewards from human feedback, such as demonstrations. However, there are often many different reward functions that explain the human feedback, leaving agents with uncertainty over what the true reward function is. While most policy optimization approaches handle this uncertainty by optimizing for expected performance, many applications demand risk-averse behavior. We derive a novel policy gradient-style robust optimization approach, PG-BROIL, that optimizes a soft-robust objective that balances expected performance and risk. To the best of our knowledge, PG-BROIL is the first policy optimization algorithm robust to a distribution of reward hypotheses which can scale to continuous MDPs. Results suggest that PG-BROIL can produce a family of behaviors ranging from risk-neutral to risk-averse and outperforms state-of-the-art imitation learning algorithms when learning from ambiguous demonstrations by hedging against uncertainty, rather than seeking to uniquely identify the demonstrator's reward function.
BibTeX
@inproceedings{javed2021policy,
  author = {Zaynah Javed and Daniel S. Brown and Satvik Sharma and Jerry Zhu and Ashwin Balakrishna and Marek Petrik and Anca D. Dragan and Ken Goldberg},
  title = {{Policy Gradient Bayesian Robust Optimization for Imitation Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=V5w1PaoVQd},
}
Figure