TL;DR
PG-BROIL is the first policy gradient algorithm for imitation learning that is robust to reward uncertainty in continuous environments. By optimizing a "soft-robust" objective, it allows agents to hedge against ambiguous human demonstrations, balancing high average performance with risk-aversion to avoid catastrophic "worst-case" outcomes.
The Challenge of Ambiguous Demonstrations
In imitation learning (IL), an agent attempts to learn a task from human demonstrations. However, these demonstrations are often ambiguous—many different reward functions could potentially explain the exact same behavior. This ambiguity creates epistemic uncertainty over the true reward function.
Standard IL methods typically handle this by picking the single most likely reward function (Maximum Likelihood or Maximum A Posteriori) or by averaging them. This approach is inherently risky: if the chosen reward function is even slightly inaccurate, the agent might engage in "reward hacking" or exhibit dangerous behaviors in unseen states. While robust methods exist to counter this, they have previously relied on linear programming, limiting them to simple, discrete "tabular" worlds. This paper bridges that gap, bringing robust imitation learning to complex, continuous environments like robotics and high-dimensional control.
Introducing PG-BROIL
To address these limitations, the authors introduce PG-BROIL (Policy Gradient Bayesian Robust Optimization for Imitation Learning). At its core is a "soft-robust" objective that balances two critical factors:
- Expected Performance: Achieving high average returns across all likely reward functions.
- Conditional Value at Risk (CVaR): Ensuring the agent avoids catastrophic failures in the "tail" of the distribution (the worst-case scenarios).
The primary technical breakthrough of PG-BROIL is the derivation of a sub-gradient for CVaR that can be seamlessly integrated into standard policy gradient frameworks like PPO or TRPO. The algorithm operates by:
- Inferring a posterior distribution of reward functions from demonstrations (using Bayesian REX).
- Evaluating policy rollouts against all sampled reward functions.
- Solving for an optimal threshold to identify which reward hypotheses the current policy is failing on.
- Re-weighting the policy update to focus more heavily on those "worst-case" rewards if the user desires risk-aversion, controlled by a tunable parameter $\lambda$.
Key Insights and Empirical Results
Instead of trying to find the "one true" reward function, PG-BROIL "hedges its bets." It finds a policy that performs reasonably well across all reward functions consistent with the demonstrations. Furthermore, while previous "risk-averse" IL methods only handled aleatoric risk (noise in environment transitions), PG-BROIL uniquely handles epistemic risk (uncertainty about the human's actual goal).
By tuning the parameter $\lambda$, the algorithm produces a family of behaviors ranging from risk-neutral (maximizing average reward) to highly risk-averse (minimizing worst-case failure). This creates a clear Pareto frontier of policies.
The researchers evaluated PG-BROIL across several continuous control domains, yielding strong results:
- Robustness to Domain Shift: In a modified Reacher task featuring a highly uncertain "red region," PG-BROIL successfully avoided the danger zone even when the goal was moved to a location unseen in demonstrations. Baseline methods frequently blundered into the red region because they overfit to a single reward hypothesis.
- Handling Ambiguity: In a custom TrashBot domain where preferences were intentionally ambiguous, PG-BROIL collected an average of 8.4 pieces of trash while almost never entering dangerous boundary regions. Baselines like GAIL and Behavioral Cloning struggled, collecting less than 3.4 pieces and frequently failing.
- High-Dimensional Success: In Atari Boxing, learning from suboptimal human demonstrations in a pixel-based environment, PG-BROIL achieved a game score of 23.9. It vastly outperformed the next best baseline (Bayesian REX at 1.6) by learning to balance scoring hits with the risk of getting hit.
Limitations and Future Work
While powerful, PG-BROIL has some limitations. The authors note that when the agent is extremely risk-averse ($\lambda$ near 0), training can become unstable due to the "jumpy" nature of the indicator function used in the CVaR gradient. Additionally, evaluating every trajectory against a large ensemble of reward functions increases the computational cost per update, though it does not require more environment interactions.
Future directions include exploring smoother risk measures (like entropic risk) to improve training stability and utilizing contrastive learning to scale the approach for learning directly from raw pixels in even more complex robotic tasks.
Summary updated