B-Pref: Benchmarking Preference-Based Reinforcement Learning

Kimin Lee, Laura Smith, Anca Dragan, Pieter Abbeel

TL;DR

This paper introduces B-Pref, a new benchmark for preference-based Reinforcement Learning (RL) that addresses the lack of standardized evaluation. B-Pref uses simulated teachers with various "irrationalities" to better reflect real-world human feedback, enabling more robust algorithm evaluation and analysis of design choices.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents learn by optimizing a reward function. However, defining accurate and comprehensive reward functions for complex tasks is notoriously difficult and time-consuming. Preference-based RL offers a solution by learning from human preferences between different agent behaviors, bypassing the need for explicit reward engineering. Despite its promise, the field of preference-based RL has lacked a standardized benchmark, making it hard to compare algorithms, quantify progress, and understand the impact of different design choices. This paper aims to fill that gap by introducing B-Pref.

Methods

The core contribution of this paper is the B-Pref benchmark. It consists of several locomotion and robotic manipulation tasks. A key innovation is the use of simulated human teachers that go beyond simply providing "perfect" preferences. These teachers are designed to exhibit various irrationalities (e.g., myopic behavior, making mistakes, or being indecisive) that are common in human decision-making. This allows for evaluating not only an algorithm's performance but also its robustness to imperfect feedback. The benchmark also proposes new evaluation metrics that account for this robustness. The authors also benchmark existing state-of-the-art preference-based RL algorithms (PrefPPO and PEBBLE) on B-Pref to demonstrate its utility.

Datasets

The B-Pref benchmark utilizes existing RL environments:

  • DeepMind Control Suite: Specifically, the Walker-walk and Quadruped-walk locomotion tasks.
  • Meta-world: Specifically, the Button Press and Sweep Into robotic manipulation tasks.

The paper does not introduce new datasets but rather a new benchmark framework for evaluating algorithms on these existing tasks.

Key Insights and Ideas

  • Realism in Simulation is Crucial: Evaluating preference-based RL algorithms solely on perfectly rational teachers is unrealistic and can mask significant weaknesses. Simulating human irrationalities is essential for developing robust agents.
  • Robustness Matters: Algorithms that perform well with ideal teachers can falter significantly when faced with imperfect or noisy feedback, highlighting the need for robustness as a key evaluation criterion.
  • Design Choices Have Varied Impacts: Different algorithmic design decisions (like how queries are selected) can have significantly different effects depending on the task and the nature of the teacher's feedback.
  • Beyond Performance: Evaluating algorithms solely on average return is insufficient. Metrics that capture robustness to irrational teachers are needed to truly understand an algorithm's practical utility.

Key Results

  • PEBBLE Outperforms PrefPPO: On average, the PEBBLE algorithm (which uses off-policy learning and unsupervised pre-training) generally outperforms PrefPPO (an on-policy method) across various environments and teacher types.
  • Irrational Teachers Expose Weaknesses: Existing methods often show a significant drop in performance when interacting with teachers exhibiting "Mistake" or "Stochastic" irrationalities, compared to an "Oracle" teacher.
  • Sampling Strategies Impact Performance: Uncertainty-based sampling schemes for selecting queries tend to yield better results than simpler uniform or coverage-based methods, even when the teacher provides more ambiguous labels.
  • Task Complexity Matters: The performance differences between algorithms and the effectiveness of certain sampling strategies can vary significantly across different task complexities (e.g., locomotion vs. robotic manipulation).

Limitations and Future Work

  • Limited Scope of Irrationalities: While B-Pref includes a range of irrationalities, it doesn't cover all possible human behaviors or combinations thereof. Future work could explore more complex teacher models.
  • Proprioceptive Inputs and Dense Rewards: The current benchmark focuses on tasks with proprioceptive inputs and dense rewards. Extending B-Pref to include visual observations and sparse rewards would be a valuable future direction.
  • Robustness to New Environments: The paper acknowledges that robustness of learned reward functions to environments with different dynamics or initial states is an important area for further research.
  • Safety Concerns: The authors briefly mention potential negative impacts, such as malicious users teaching undesirable behaviors, suggesting that safety considerations are important for future work in this area.
  • New Algorithms: The observed performance gaps with irrational teachers suggest a need for developing new preference-based RL algorithms that are inherently more robust.

Summary updated

Abstract

Reinforcement learning (RL) requires access to a reward function that incentivizes the right behavior, but these are notoriously hard to specify for complex tasks. Preference-based RL provides an alternative: learning policies using a teacher's preferences without pre-defined rewards, thus overcoming concerns associated with reward engineering. However, it is difficult to quantify the progress in preference-based RL due to the lack of a commonly adopted benchmark. In this paper, we introduce B-Pref: a benchmark specially designed for preference-based RL. A key challenge with such a benchmark is providing the ability to evaluate candidate algorithms quickly, which makes relying on real human input for evaluation prohibitive. At the same time, simulating human input as giving perfect preferences for the ground truth reward function is unrealistic. B-Pref alleviates this by simulating teachers with a wide array of irrationalities, and proposes metrics not solely for performance but also for robustness to these potential irrationalities. We showcase the utility of B-Pref by using it to analyze algorithmic design choices, such as selecting informative queries, for state-of-the-art preference-based RL algorithms. We hope that B-Pref can serve as a common starting point to study preference-based RL more systematically. Source code is available at https://github.com/rll-research/B-Pref.
BibTeX
@inproceedings{lee2021b,
  author = {Kimin Lee and Laura Smith and Anca Dragan and Pieter Abbeel},
  title = {{B-Pref: Benchmarking Preference-Based Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=ps95-mkHF_},
}
Figure