TL;DR
This paper introduces a novel framework for deep reinforcement learning (RL) that uses "weak" human preferences, allowing for more nuanced feedback than simple "better/worse" choices. By developing a preference scaling model and a human-demonstration estimator, they significantly reduce the amount of human input needed, achieving comparable or better performance in robot locomotion tasks.
What Problem Does This Paper Address?
Reinforcement learning (RL) agents learn by maximizing rewards, but defining an accurate reward function for complex tasks can be incredibly difficult. Traditional RL often leads to behaviors that don't align with human expectations. While "RL from human preferences" (RLHP) offers a solution by learning from pairwise comparisons of robot trajectories, it has limitations: it requires a massive amount of human input (thousands of judgments) and only accepts discrete preferences (left, right, or equal). This is both time-consuming for humans and doesn't capture the subtle degrees of preference humans naturally express. The paper aims to address this by enabling more flexible, "weak" human preferences and reducing the overall human effort required.
Methods
The core of the proposed solution lies in two key innovations:
- Weak Human Preferences: Synthetic Preference Scaling: Instead of just "left," "right," or "equal," the authors introduce a "synthetic oracle" that mimics human preference. This oracle assigns a continuous score between 0.0 and 1.0 to represent the degree of preference. A score of 1.0 means absolute preference for the left trajectory, 0.0 for the right, and 0.5 for indifference. Values in between represent "weak" preferences, capturing nuances like "slightly prefer left." This scaling is normalized based on a distribution of rewards to ensure robustness.
- Human-Demonstration Estimator: To further reduce human input, the authors developed a supervised learning model (using linear regression or Support Vector Regression) that acts as a "human-demonstration estimator." This model learns to predict human preferences based on a smaller initial set of human judgments. This allows the RL agent to generate a significant portion of the preference labels itself, drastically cutting down on the need for direct human feedback.
These two components are integrated into a framework called "Weak Human Preference Supervision for Deep Reinforcement Learning" (RLHPS), and a variant that uses the estimator is called RLHPS with Demo.
Datasets
The experiments were conducted using the MuJoCo physics engine within the OpenAI Gym platform. Five specific MuJoCo scenarios were used for evaluation: Walker, Hopper, Swimmer, Cheetah, and Ant. These are continuous control tasks that simulate robot locomotion. The paper does not introduce new datasets but utilizes existing benchmarks for evaluating RL agents.
Key Insights and Ideas
- Nuance Matters: Human preferences aren't always binary. Allowing for "weak" preferences (e.g., a score of 0.7 instead of just "left") provides richer information to the RL agent, leading to better alignment with human intent.
- Human Effort is a Bottleneck: The sheer volume of human input required by existing RLHP methods is a major barrier. Reducing this burden is crucial for practical applications.
- Prediction is Powerful: A supervised learning model can effectively learn to predict human preferences, significantly reducing the need for direct human interaction. This "human-demonstration estimator" acts as a valuable proxy.
- Trade-offs Exist: While reducing human input is beneficial, there's a point where too much estimation can negatively impact performance. The paper highlights that a 30% reduction in human input is effective, but a 50% reduction leads to a noticeable drop in performance.
Key Results
- Higher Cumulative Rewards: The proposed RLHPS method consistently achieved higher cumulative reward values compared to traditional RL (PPO) and the standard RLHP method across most scenarios. For instance, in the Walker scenario with 1,400 human inputs, RLHPS achieved approximately 1,500 higher rewards than PPO and 1,000 higher than RLHP.
- Significant Reduction in Human Input: The RLHPS with Demo approach reduced human input by up to 30%. This means that instead of needing 1,400 labels, only about 980 were required, saving approximately 7 minutes per experiment in MuJoCo.
- Comparable Performance with Reduced Input: With a 30% reduction in human input (RLHPS with Demo (30%)), the performance was similar or even superior to RLHPS without the estimator. However, a 50% reduction (RLHPS with Demo (50%)) led to performance comparable to PPO, indicating the limit of estimation.
- Improved Training Dynamics: Instant reward distributions showed that RLHPS leads to more favorable rewards at the end of training compared to the initial stages, confirming the positive impact of the learned preferences.
Limitations and Future Work
- Frame Selection: The current method selects frames for human comparison randomly. The authors suggest that an intelligent selection model could improve this by choosing segments with diverse rewards or where the prediction model is uncertain.
- Normalisation Sensitivity: The normalisation step of the human preference scaling model could be further investigated, as reward distributions might influence preference levels for the estimator.
- Extent of Estimation: The paper demonstrates that while a 30% reduction in human input is effective, a 50% reduction significantly impacts performance. Future work could explore more sophisticated estimation techniques or hybrid approaches to push this boundary further.
- Generalizability: While tested on MuJoCo, further validation on other complex RL domains would be beneficial.
Summary updated