Policy Gradient Bayesian Robust Optimization for Imitation Learning

Zaynah Javed, Daniel S. Brown, Satvik Sharma, Jerry Zhu, Ashwin Balakrishna, Marek Petrik, Anca D. Dragan, Ken Goldberg

TL;DR

PG-BROIL is the first policy gradient algorithm for imitation learning that is robust to reward uncertainty in continuous environments. By optimizing a "soft-robust" objective, it allows agents to hedge against ambiguous human demonstrations, balancing high average performance with risk-aversion to avoid catastrophic "worst-case" outcomes.

The Challenge of Ambiguous Demonstrations

In imitation learning (IL), an agent attempts to learn a task from human demonstrations. However, these demonstrations are often ambiguous—many different reward functions could potentially explain the exact same behavior. This ambiguity creates epistemic uncertainty over the true reward function.

Standard IL methods typically handle this by picking the single most likely reward function (Maximum Likelihood or Maximum A Posteriori) or by averaging them. This approach is inherently risky: if the chosen reward function is even slightly inaccurate, the agent might engage in "reward hacking" or exhibit dangerous behaviors in unseen states. While robust methods exist to counter this, they have previously relied on linear programming, limiting them to simple, discrete "tabular" worlds. This paper bridges that gap, bringing robust imitation learning to complex, continuous environments like robotics and high-dimensional control.

Introducing PG-BROIL

To address these limitations, the authors introduce PG-BROIL (Policy Gradient Bayesian Robust Optimization for Imitation Learning). At its core is a "soft-robust" objective that balances two critical factors:

  1. Expected Performance: Achieving high average returns across all likely reward functions.
  2. Conditional Value at Risk (CVaR): Ensuring the agent avoids catastrophic failures in the "tail" of the distribution (the worst-case scenarios).
CVaR measures the average of the worst outcomes in the distribution's tail.

The primary technical breakthrough of PG-BROIL is the derivation of a sub-gradient for CVaR that can be seamlessly integrated into standard policy gradient frameworks like PPO or TRPO. The algorithm operates by:

  • Inferring a posterior distribution of reward functions from demonstrations (using Bayesian REX).
  • Evaluating policy rollouts against all sampled reward functions.
  • Solving for an optimal threshold to identify which reward hypotheses the current policy is failing on.
  • Re-weighting the policy update to focus more heavily on those "worst-case" rewards if the user desires risk-aversion, controlled by a tunable parameter $\lambda$.

Key Insights and Empirical Results

Instead of trying to find the "one true" reward function, PG-BROIL "hedges its bets." It finds a policy that performs reasonably well across all reward functions consistent with the demonstrations. Furthermore, while previous "risk-averse" IL methods only handled aleatoric risk (noise in environment transitions), PG-BROIL uniquely handles epistemic risk (uncertainty about the human's actual goal).

By tuning the parameter $\lambda$, the algorithm produces a family of behaviors ranging from risk-neutral (maximizing average reward) to highly risk-averse (minimizing worst-case failure). This creates a clear Pareto frontier of policies.

Experimental domains and the Pareto frontier demonstrating the trade-off between expected return and robustness.

The researchers evaluated PG-BROIL across several continuous control domains, yielding strong results:

  • Robustness to Domain Shift: In a modified Reacher task featuring a highly uncertain "red region," PG-BROIL successfully avoided the danger zone even when the goal was moved to a location unseen in demonstrations. Baseline methods frequently blundered into the red region because they overfit to a single reward hypothesis.
  • Handling Ambiguity: In a custom TrashBot domain where preferences were intentionally ambiguous, PG-BROIL collected an average of 8.4 pieces of trash while almost never entering dangerous boundary regions. Baselines like GAIL and Behavioral Cloning struggled, collecting less than 3.4 pieces and frequently failing.
  • High-Dimensional Success: In Atari Boxing, learning from suboptimal human demonstrations in a pixel-based environment, PG-BROIL achieved a game score of 23.9. It vastly outperformed the next best baseline (Bayesian REX at 1.6) by learning to balance scoring hits with the risk of getting hit.

Limitations and Future Work

While powerful, PG-BROIL has some limitations. The authors note that when the agent is extremely risk-averse ($\lambda$ near 0), training can become unstable due to the "jumpy" nature of the indicator function used in the CVaR gradient. Additionally, evaluating every trajectory against a large ensemble of reward functions increases the computational cost per update, though it does not require more environment interactions.

Future directions include exploring smoother risk measures (like entropic risk) to improve training stability and utilizing contrastive learning to scale the approach for learning directly from raw pixels in even more complex robotic tasks.

Summary updated

Abstract

The difficulty in specifying rewards for many real-world problems has led to an increased focus on learning rewards from human feedback, such as demonstrations. However, there are often many different reward functions that explain the human feedback, leaving agents with uncertainty over what the true reward function is. While most policy optimization approaches handle this uncertainty by optimizing for expected performance, many applications demand risk-averse behavior. We derive a novel policy gradient-style robust optimization approach, PG-BROIL, that optimizes a soft-robust objective that balances expected performance and risk. To the best of our knowledge, PG-BROIL is the first policy optimization algorithm robust to a distribution of reward hypotheses which can scale to continuous MDPs. Results suggest that PG-BROIL can produce a family of behaviors ranging from risk-neutral to risk-averse and outperforms state-of-the-art imitation learning algorithms when learning from ambiguous demonstrations by hedging against uncertainty, rather than seeking to uniquely identify the demonstrator’s reward function.
BibTeX
@inproceedings{javed2021policy,
  author = {Zaynah Javed and Daniel S. Brown and Satvik Sharma and Jerry Zhu and Ashwin Balakrishna and Marek Petrik and Anca D. Dragan and Ken Goldberg},
  title = {{Policy Gradient Bayesian Robust Optimization for Imitation Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=V5w1PaoVQd},
}
Figure