A Survey of Preference-Based Reinforcement Learning Methods

Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes FürnkranzJMLR 2017 · OpenReview

TL;DR

This paper surveys Preference-Based Reinforcement Learning (PbRL), a paradigm that learns from expert preferences instead of explicit reward functions. It offers a unified framework for understanding PbRL, categorizing its design principles, and highlighting its potential to simplify RL application in complex domains by reducing reliance on expert knowledge and manual reward engineering.

What Problem Does This Paper Address?

Traditional Reinforcement Learning (RL) relies on carefully designed reward functions to guide agents. However, crafting these functions is often challenging, requiring significant domain expertise and leading to issues like "reward hacking" (agents exploiting loopholes in the reward function) or "reward shaping" (providing intermediate rewards that might not align with the true goal). These problems are particularly acute in real-world robotics and complex tasks where defining a precise numerical reward is difficult or impossible. PbRL aims to overcome these limitations by learning directly from an expert's comparative judgments (preferences) between different behaviors, rather than requiring a numerical reward signal.

Methods

The paper presents a unified framework for PbRL, breaking down the problem into key design choices:

  • Types of Preferences: Trajectory, action, or state preferences, each with varying levels of complexity for the expert and algorithm.
  • Learning Problem Formulation:
    • Direct Policy Learning: Directly optimizing a policy to align with preferences.
  • Surrogate-Based Learning: Learning an intermediate model (preference model or utility function) that approximates the expert's preferences, which is then used to derive a policy.
  • Preference Elicitation: Strategies for generating informative queries to the expert and exploring the system dynamics. This includes undirected exploration (random sampling) and directed exploration (guided by uncertainty reduction).
  • Optimization Strategies: Methods for optimizing policies based on learned preferences or utility functions, including policy search and value-based approaches.
  • Transition Dynamics Modeling: Discusses model-based (learning system dynamics) and model-free (no explicit model) approaches.

Datasets

Not explicitly discussed in the paper, but the authors mention that many approaches are evaluated on classic RL toy domains like mountain car, pendulum, cart-pole, and acrobot. They also highlight applications in robotics (object handover, path planning) and games (chess, Othello).

Key Insights and Ideas

  • Preference Learning as a Solution to Reward Engineering: PbRL offers a powerful alternative to manual reward design, making RL more accessible for complex, real-world tasks.
  • Unified Framework: The paper provides a much-needed structured way to understand the diverse landscape of PbRL algorithms by categorizing them based on core design principles.
  • Trade-offs in Preference Types: Different preference types (trajectory, action, state) offer varying levels of information and complexity, requiring careful consideration based on the task and expert's capabilities.
  • Exploration is Crucial: Effectively exploring both the policy space and the preference space is essential for efficient learning and reducing the number of expert queries.
  • Surrogate Models Simplify Optimization: Learning an intermediate preference model or utility function can leverage existing RL optimization techniques, making the learning process more tractable.

Key Results

The paper is a survey, so it doesn't present new empirical results. Instead, it synthesizes findings from numerous existing works. It highlights that PbRL has been successfully applied to various domains, including robotics (object handover, path planning) and games (chess), often achieving performance comparable to or exceeding handcrafted solutions. The effectiveness of PbRL is demonstrated by its ability to learn complex behaviors without explicit reward functions.

Limitations and Future Work

  • Sample Efficiency: Many PbRL approaches still require a high number of preferences, which can be costly to obtain from human experts.
  • Preference Space Exploration: Principled integration of preference-space exploration with policy-space exploration is still an open area.
  • Handling Incomparabilities: Current methods often struggle with incomparable trajectories, and learning non-scalar utility functions is an avenue for future research.
  • Generalization to Complex Tasks: Applying PbRL to tasks requiring deep neural networks for sensory input processing is an emerging area.
  • Unified Evaluation Framework: The lack of standardized benchmarks and evaluation protocols makes direct comparison of algorithms challenging.
  • Risk-Averse Optimization: Extending PbRL to handle risk-averse scenarios (e.g., avoiding dangerous trajectories) is an important future direction.
  • Combining Feedback Types: Integrating preferences with other feedback forms like demonstrations or ordinal/numeric feedback could further reduce cognitive load.
  • Handling Different Initial States: Developing methods that can handle preferences between trajectories starting from different initial states is a significant challenge.

Summary updated

Abstract

Reinforcement learning (RL) techniques optimize the accumulated long-term reward of a suitably chosen reward function. However, designing such a reward function often requires a lot of task- specific prior knowledge. The designer needs to consider different objectives that do not only influence the learned behavior but also the learning progress. To alleviate these issues, preference-based reinforcement learning algorithms (PbRL) have been proposed that can directly learn from an expert's preferences instead of a hand-designed numeric reward. PbRL has gained traction in recent years due to its ability to resolve the reward shaping problem, its ability to learn from non numeric rewards and the possibility to reduce the dependence on expert knowledge. We provide a unified framework for PbRL that describes the task formally and points out the different design principles that affect the evaluation task for the human as well as the computational complexity. The design principles include the type of feedback that is assumed, the representation that is learned to capture the preferences, the optimization problem that has to be solved as well as how the exploration/exploitation problem is tackled. Furthermore, we point out shortcomings of current algorithms, propose open research questions and briefly survey practical tasks that have been solved using PbRL.
BibTeX
@article{wirth2017survey,
  author = {Christian Wirth and Riad Akrour and Gerhard Neumann and Johannes Fürnkranz},
  title = {{A Survey of Preference-Based Reinforcement Learning Methods}},
  journal = {J. Mach. Learn. Res.},
  year = {2017},
  url = {https://openreview.net/forum?id=DlqcKH4cjUl},
}
Figure