Abstract
Reinforcement learning (RL) requires access to a reward function that incentivizes the right behavior, but these are notoriously hard to specify for complex tasks. Preference-based RL provides an alternative: learning policies using a teacher's preferences without pre-defined rewards, thus overcoming concerns associated with reward engineering. However, it is difficult to quantify the progress in preference-based RL due to the lack of a commonly adopted benchmark. In this paper, we introduce B-Pref: a benchmark specially designed for preference-based RL. A key challenge with such a benchmark is providing the ability to evaluate candidate algorithms quickly, which makes relying on real human input for evaluation prohibitive. At the same time, simulating human input as giving perfect preferences for the ground truth reward function is unrealistic. B-Pref alleviates this by simulating teachers with a wide array of irrationalities, and proposes metrics not solely for performance but also for robustness to these potential irrationalities. We showcase the utility of B-Pref by using it to analyze algorithmic design choices, such as selecting informative queries, for state-of-the-art preference-based RL algorithms. We hope that B-Pref can serve as a common starting point to study preference-based RL more systematically.
Introduction
In traditional Reinforcement Learning (RL), agents are guided by a reward function. However, designing these functions is notoriously difficult, often leading to "reward hacking" where agents exploit poorly specified rewards without achieving the intended goal. Preference-based RL (PbRL) offers a promising alternative by allowing a human teacher to provide feedback through simple pairwise comparisons of agent behavior clips.
Despite its potential, evaluating PbRL algorithms remains challenging. Real-world human feedback is slow and expensive, while current simulations often assume a "perfect" teacher—an unrealistic premise given human noise, myopia, and error. B-Pref addresses this gap by providing a standardized benchmark that incorporates simulated "irrational" human behaviors.
The B-Pref Benchmark
B-Pref introduces a SimTeacher capable of exhibiting various human-like irrationalities to test the robustness of PbRL agents:
- Stochasticity: Noisy decision-making where the better clip isn't always chosen.
- Mistakes: Occasional accidental flipping of preferences.
- Myopia: A bias toward the end of a trajectory, reflecting short-term memory.
- Non-forced Choice: Options for the teacher to "skip" queries or mark them as "equal," avoiding forced preferences between indistinguishable behaviors.
The benchmark utilizes tasks from the DeepMind Control Suite (Walker, Quadruped) and Meta-world (Button Press, Sweep Into). Performance is measured using Normalized Returns, comparing preference-learned policies against those trained with ground-truth rewards.
Experimental Analysis
The authors benchmarked two state-of-the-art algorithms: PrefPPO (on-policy) and PEBBLE (off-policy with unsupervised pre-training).
Key Insights
- Robustness to Noise: Current algorithms are surprisingly fragile to "Mistake" teachers (10% label flip), often failing to reach 50% of the performance achieved with an "Oracle" teacher.
- The Benefit of Non-Forced Choice: Allowing teachers to skip or mark clips as equal often improved performance on complex tasks like the Quadruped, as it prevents the injection of noise from forced choices between poor behaviors.
- Query Selection: Uncertainty-based sampling, which prioritizes clips where the agent's reward model is most confused, significantly outperformed random sampling.
- Algorithm Comparison: PEBBLE generally outperformed PrefPPO, particularly in manipulation tasks, due to its efficient use of off-policy data and unsupervised exploration.
Conclusion
B-Pref highlights a significant "robustness gap" in current PbRL research, particularly regarding human error and label corruption. While existing methods handle myopia and non-forced choices relatively well, they struggle with stochastic mistakes. By providing a standardized evaluation framework with simulated human messiness, B-Pref aims to catalyze the development of more robust and practical preference-based RL algorithms. Future work will focus on expanding the benchmark to include raw pixel inputs and more complex, multi-faceted teacher models.
Summary updated