Weak Human Preference Supervision for Deep Reinforcement Learning

Zehong Cao, Kaichiu Wong, Chin-Teng LinIEEE TNNLS 2021 · DOI · OpenReview

TL;DR

This paper introduces a novel framework for deep reinforcement learning (RL) that uses "weak" human preferences, allowing for more nuanced feedback than simple "better/worse" choices. By developing a preference scaling model and a human-demonstration estimator, they significantly reduce the amount of human input needed, achieving comparable or better performance in robot locomotion tasks.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents learn by maximizing rewards, but defining an accurate reward function for complex tasks can be incredibly difficult. Traditional RL often leads to behaviors that don't align with human expectations. While "RL from human preferences" (RLHP) offers a solution by learning from pairwise comparisons of robot trajectories, it has limitations: it requires a massive amount of human input (thousands of judgments) and only accepts discrete preferences (left, right, or equal). This is both time-consuming for humans and doesn't capture the subtle degrees of preference humans naturally express. The paper aims to address this by enabling more flexible, "weak" human preferences and reducing the overall human effort required.

Methods

The core of the proposed solution lies in two key innovations:

  1. Weak Human Preferences: Synthetic Preference Scaling: Instead of just "left," "right," or "equal," the authors introduce a "synthetic oracle" that mimics human preference. This oracle assigns a continuous score between 0.0 and 1.0 to represent the degree of preference. A score of 1.0 means absolute preference for the left trajectory, 0.0 for the right, and 0.5 for indifference. Values in between represent "weak" preferences, capturing nuances like "slightly prefer left." This scaling is normalized based on a distribution of rewards to ensure robustness.
  2. Human-Demonstration Estimator: To further reduce human input, the authors developed a supervised learning model (using linear regression or Support Vector Regression) that acts as a "human-demonstration estimator." This model learns to predict human preferences based on a smaller initial set of human judgments. This allows the RL agent to generate a significant portion of the preference labels itself, drastically cutting down on the need for direct human feedback.

These two components are integrated into a framework called "Weak Human Preference Supervision for Deep Reinforcement Learning" (RLHPS), and a variant that uses the estimator is called RLHPS with Demo.

Datasets

The experiments were conducted using the MuJoCo physics engine within the OpenAI Gym platform. Five specific MuJoCo scenarios were used for evaluation: Walker, Hopper, Swimmer, Cheetah, and Ant. These are continuous control tasks that simulate robot locomotion. The paper does not introduce new datasets but utilizes existing benchmarks for evaluating RL agents.

Key Insights and Ideas

  • Nuance Matters: Human preferences aren't always binary. Allowing for "weak" preferences (e.g., a score of 0.7 instead of just "left") provides richer information to the RL agent, leading to better alignment with human intent.
  • Human Effort is a Bottleneck: The sheer volume of human input required by existing RLHP methods is a major barrier. Reducing this burden is crucial for practical applications.
  • Prediction is Powerful: A supervised learning model can effectively learn to predict human preferences, significantly reducing the need for direct human interaction. This "human-demonstration estimator" acts as a valuable proxy.
  • Trade-offs Exist: While reducing human input is beneficial, there's a point where too much estimation can negatively impact performance. The paper highlights that a 30% reduction in human input is effective, but a 50% reduction leads to a noticeable drop in performance.

Key Results

  • Higher Cumulative Rewards: The proposed RLHPS method consistently achieved higher cumulative reward values compared to traditional RL (PPO) and the standard RLHP method across most scenarios. For instance, in the Walker scenario with 1,400 human inputs, RLHPS achieved approximately 1,500 higher rewards than PPO and 1,000 higher than RLHP.
  • Significant Reduction in Human Input: The RLHPS with Demo approach reduced human input by up to 30%. This means that instead of needing 1,400 labels, only about 980 were required, saving approximately 7 minutes per experiment in MuJoCo.
  • Comparable Performance with Reduced Input: With a 30% reduction in human input (RLHPS with Demo (30%)), the performance was similar or even superior to RLHPS without the estimator. However, a 50% reduction (RLHPS with Demo (50%)) led to performance comparable to PPO, indicating the limit of estimation.
  • Improved Training Dynamics: Instant reward distributions showed that RLHPS leads to more favorable rewards at the end of training compared to the initial stages, confirming the positive impact of the learned preferences.

Limitations and Future Work

  • Frame Selection: The current method selects frames for human comparison randomly. The authors suggest that an intelligent selection model could improve this by choosing segments with diverse rewards or where the prediction model is uncertain.
  • Normalisation Sensitivity: The normalisation step of the human preference scaling model could be further investigated, as reward distributions might influence preference levels for the estimator.
  • Extent of Estimation: The paper demonstrates that while a 30% reduction in human input is effective, a 50% reduction significantly impacts performance. Future work could explore more sophisticated estimation techniques or hybrid approaches to push this boundary further.
  • Generalizability: While tested on MuJoCo, further validation on other complex RL domains would be beneficial.

Summary updated

Abstract

The current reward learning from human preferences could be used to resolve complex reinforcement learning (RL) tasks without access to a reward function by defining a single fixed preference between pairs of trajectory segments. However, the judgment of preferences between trajectories is not dynamic and still requires human input over thousands of iterations. In this study, we proposed a weak human preference supervision framework, for which we developed a human preference scaling model that naturally reflects the human perception of the degree of weak choices between trajectories and established a human-demonstration estimator through supervised learning to generate the predicted preferences for reducing the number of human inputs. The proposed weak human preference supervision framework can effectively solve complex RL tasks and achieve higher cumulative rewards in simulated robot locomotion—MuJoCo games—relative to the single fixed human preferences. Furthermore, our established human-demonstration estimator requires human feedback only for less than 0.01% of the agent’s interactions with the environment and significantly reduces the cost of human inputs by up to 30% compared with the existing approaches. To present the flexibility of our approach, we released a video ( <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://youtu.be/jQPe1OILT0M</uri> ) showing comparisons of the behaviors of agents trained on different types of human input. We believe that our naturally inspired human preferences with weakly supervised learning are beneficial for precise reward learning and can be applied to state-of-the-art RL systems, such as human-autonomy teaming systems.
BibTeX
@article{cao2021weak,
  author = {Zehong Cao and Kaichiu Wong and Chin-Teng Lin},
  title = {{Weak Human Preference Supervision for Deep Reinforcement Learning}},
  journal = {IEEE TNNLS},
  year = {2021},
  doi = {10.1109/tnnls.2021.3084198},
}
Figure