Abstract
Reinforcement Learning from Human Preferences (RLHP) is a promising technique for training agents in environments where reward functions are difficult to specify. However, standard RLHP suffers from feedback rigidity and high human labor costs. This paper proposes a "weak human preference" framework that utilizes continuous preference scaling and a supervised learning estimator to predict human choices. Experimental results on MuJoCo benchmarks demonstrate that this approach achieves higher cumulative rewards while reducing human effort by up to 30%.
1. Introduction
Reinforcement Learning (RL) typically relies on hand-crafted reward functions, which are often prone to "reward hacking"—where agents exploit mathematical loopholes rather than achieving the intended goal. While learning from human preferences (RLHP) mitigates this, existing systems are often limited by binary choice models (Left, Right, or Equal) and the high cost of human expert time. This research introduces a framework to address these challenges by incorporating nuanced preference scaling and automated preference estimation.
2. Methodology
The proposed system enhances the standard RLHP pipeline through two primary innovations: Synthetic Preference Scaling and a Human-Demonstration Estimator.
2.1 Synthetic Preference Scaling (RLHPS)
Traditional RLHP uses discrete labels. The authors propose a scaling model that assigns a continuous value $z \in [0, 1]$ to trajectory pairs. This "weak preference" model captures the degree to which one trajectory is superior to another, providing a much denser and more informative signal for the reward predictor.
2.2 Human-Demonstration Estimator
To reduce human workload, the authors treat preference prediction as a supervised regression problem. After an initial set of human labels is collected, a model—either Linear Regression or Support Vector Regression (SVR)—is trained to act as a "demonstration estimator." This model predicts preferences for 30–50% of the remaining trajectory pairs, effectively substituting for human input.
3. Experimental Setup
The framework was evaluated using the MuJoCo physics engine across several continuous control benchmarks: Walker, Hopper, Swimmer, Cheetah, and Ant. The underlying RL agent was trained using Proximal Policy Optimization (PPO).
4. Results and Discussion
The RLHPS framework demonstrated significant performance gains over both standard PPO and traditional RLHP. In the Walker scenario, RLHPS achieved a cumulative reward approximately 1,500 points higher than PPO and 1,000 points higher than standard RLHP.
Key findings include:
- Efficiency: The system requires human feedback for less than 0.01% of the agent's total environment interactions.
- Effort Reduction: The demonstration estimator successfully reduced human input by 30% (from 1,400 to 980 labels) without a significant drop in performance.
- Robustness: In complex 3D environments like "Ant," the continuous scale allowed the agent to learn stable walking patterns where fixed-preference models failed.
5. Conclusion and Future Work
This work demonstrates that continuous preference scaling and supervised estimation can significantly improve the efficiency and performance of RLHP. However, limitations remain; pushing the estimator to predict 50% of labels resulted in increased error and performance degradation. Future research will explore active learning techniques to intelligently select trajectory segments for human review, potentially further reducing the labeling burden.
Summary updated