Human Preference Scaling with Demonstrations For Deep Reinforcement Learning

Zehong Cao, Kaichiu Wong, Chin-Teng LinarXiv 2020 · arXiv

Abstract

Reinforcement Learning from Human Preferences (RLHP) is a promising technique for training agents in environments where reward functions are difficult to specify. However, standard RLHP suffers from feedback rigidity and high human labor costs. This paper proposes a "weak human preference" framework that utilizes continuous preference scaling and a supervised learning estimator to predict human choices. Experimental results on MuJoCo benchmarks demonstrate that this approach achieves higher cumulative rewards while reducing human effort by up to 30%.

1. Introduction

Reinforcement Learning (RL) typically relies on hand-crafted reward functions, which are often prone to "reward hacking"—where agents exploit mathematical loopholes rather than achieving the intended goal. While learning from human preferences (RLHP) mitigates this, existing systems are often limited by binary choice models (Left, Right, or Equal) and the high cost of human expert time. This research introduces a framework to address these challenges by incorporating nuanced preference scaling and automated preference estimation.

2. Methodology

The proposed system enhances the standard RLHP pipeline through two primary innovations: Synthetic Preference Scaling and a Human-Demonstration Estimator.

2.1 Synthetic Preference Scaling (RLHPS)

Traditional RLHP uses discrete labels. The authors propose a scaling model that assigns a continuous value $z \in [0, 1]$ to trajectory pairs. This "weak preference" model captures the degree to which one trajectory is superior to another, providing a much denser and more informative signal for the reward predictor.

2.2 Human-Demonstration Estimator

To reduce human workload, the authors treat preference prediction as a supervised regression problem. After an initial set of human labels is collected, a model—either Linear Regression or Support Vector Regression (SVR)—is trained to act as a "demonstration estimator." This model predicts preferences for 30–50% of the remaining trajectory pairs, effectively substituting for human input.

Comparison of standard preference scaling (A) and the proposed RLHPS with a Preference Estimator (B).

3. Experimental Setup

The framework was evaluated using the MuJoCo physics engine across several continuous control benchmarks: Walker, Hopper, Swimmer, Cheetah, and Ant. The underlying RL agent was trained using Proximal Policy Optimization (PPO).

The Walker environment, one of the primary benchmarks used for evaluation.

4. Results and Discussion

The RLHPS framework demonstrated significant performance gains over both standard PPO and traditional RLHP. In the Walker scenario, RLHPS achieved a cumulative reward approximately 1,500 points higher than PPO and 1,000 points higher than standard RLHP.

Key findings include:

  • Efficiency: The system requires human feedback for less than 0.01% of the agent's total environment interactions.
  • Effort Reduction: The demonstration estimator successfully reduced human input by 30% (from 1,400 to 980 labels) without a significant drop in performance.
  • Robustness: In complex 3D environments like "Ant," the continuous scale allowed the agent to learn stable walking patterns where fixed-preference models failed.

5. Conclusion and Future Work

This work demonstrates that continuous preference scaling and supervised estimation can significantly improve the efficiency and performance of RLHP. However, limitations remain; pushing the estimator to predict 50% of labels resulted in increased error and performance degradation. Future research will explore active learning techniques to intelligently select trajectory segments for human review, potentially further reducing the labeling burden.

Summary updated

Abstract

The current reward learning from human preferences could be used to resolve complex reinforcement learning (RL) tasks without access to a reward function by defining a single fixed preference between pairs of trajectory segments. However, the judgement of preferences between trajectories is not dynamic and still requires human input over thousands of iterations. In this study, we proposed a weak human preference supervision framework, for which we developed a human preference scaling model that naturally reflects the human perception of the degree of weak choices between trajectories and established a human-demonstration estimator via supervised learning to generate the predicted preferences for reducing the number of human inputs. The proposed weak human preference supervision framework can effectively solve complex RL tasks and achieve higher cumulative rewards in simulated robot locomotion -- MuJoCo games -- relative to the single fixed human preferences. Furthermore, our established human-demonstration estimator requires human feedback only for less than 0.01\% of the agent's interactions with the environment and significantly reduces the cost of human inputs by up to 30\% compared with the existing approaches. To present the flexibility of our approach, we released a video (https://youtu.be/jQPe1OILT0M) showing comparisons of the behaviours of agents trained on different types of human input. We believe that our naturally inspired human preferences with weakly supervised learning are beneficial for precise reward learning and can be applied to state-of-the-art RL systems, such as human-autonomy teaming systems.
BibTeX
@misc{cao2020human,
  author = {Zehong Cao and Kaichiu Wong and Chin-Teng Lin},
  title = {{Human Preference Scaling with Demonstrations For Deep Reinforcement Learning}},
  year = {2020},
  eprint = {2007.12904},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.12904},
}
Figure