Introduction
Reinforcement Learning (RL) has traditionally relied on meticulously hand-crafted reward functions to guide agent behavior. However, for complex manipulation tasks, designing these functions is often difficult and prone to "reward hacking," where agents exploit flaws in the reward definition rather than achieving the intended goal. Preference-based Reinforcement Learning (PbRL) addresses this by allowing human teachers to provide feedback through simple binary comparisons of robot trajectories.
Despite its promise, PbRL faces two major hurdles: feedback inefficiency and sample inefficiency. Humans are often asked to compare redundant or uninformative segments, leading to noisy labels and slow convergence. Furthermore, standard exploration strategies often fail to guide the agent toward task-relevant regions of the state space. This paper introduces SENIOR (efficient query Selection and preferENce-guIded explORation), a framework designed to optimize both human feedback and agent exploration.
The SENIOR Framework
SENIOR integrates two novel components to streamline the learning process: Motion-Distinction-based Selection (MDS) and Preference-Guided Exploration (PGE).
Motion-Distinction-based Selection (MDS)
MDS improves feedback efficiency by ensuring that the queries presented to humans are easy to interpret and highly informative. Instead of random selection, MDS uses Kernel Density Estimation (KDE) to identify segments where the robot exhibits significant motion (low state density). It then applies Principal Component Analysis (PCA) to determine the primary direction of this motion. By prioritizing pairs of segments with distinct motions in different directions, MDS reduces the cognitive load on the human teacher and improves the accuracy of the learned reward model.
Preference-Guided Exploration (PGE)
To address sample inefficiency, PGE leverages the learned preference model to guide the agent's exploration. It introduces a "curiosity" reward that encourages the robot to visit states that are predicted to be highly preferred by humans but have been rarely visited by the agent. This creates a synergistic loop where human feedback directly informs the agent's exploration strategy, pulling it toward the goal more effectively than random or purely uncertainty-based exploration.
Experimental Evaluation
The authors evaluated SENIOR on the Meta-World benchmark, focusing on six challenging manipulation tasks: Door Lock, Window Close, Handle Press, Window Open, Door Open, and Door Unlock. They also demonstrated the framework's robustness through real-world deployment on a UR5 robotic arm.
Key Results
- Performance Gains: The "M-SENIOR" variant (built on Meta-Reward-Net) achieved a 97% average success rate across all Meta-World tasks, significantly outperforming baselines like PEBBLE (53%) and M-RUNE (70%).
- Feedback Efficiency: SENIOR demonstrated remarkable efficiency, achieving near-perfect success on the Door Lock task with only 250 feedback labels, whereas competing methods required up to 1000 labels to reach lower performance levels.
- Real-World Transfer: Policies learned via SENIOR successfully transferred to a physical UR5 robot, maintaining high success rates (85-100%) on tasks like opening and closing doors and boxes.
Discussion and Future Work
SENIOR demonstrates that human-centric design—focusing on interpretability and direct guidance—can drastically improve the efficiency of PbRL. By making queries "easy-to-compare" and using preferences to guide exploration, the framework bridges the gap between human intuition and robotic execution.
Future research directions include integrating Large Language Models (LLMs) to provide prior task knowledge, potentially reducing the need for initial human feedback. Additionally, expanding the motion-based selection criteria to encompass more abstract state features could broaden the framework's applicability to non-manipulation tasks.
Summary updated