Abstract
Conveying complex objectives to reinforcement learning (RL) agents can often be difficult, involving meticulous design of reward functions that are sufficiently informative yet easy enough to provide. Human-in-the-loop RL methods allow practitioners to instead interactively teach agents through tailored feedback; however, such approaches have been challenging to scale since human feedback is very expensive. In this work, we aim to make this process more sample- and feedback-efficient. We present an off-policy, interactive RL algorithm that capitalizes on the strengths of both feedback and off-policy learning. Specifically, we learn a reward model by actively querying a teacher's preferences between two clips of behavior and use it to train an agent. To enable off-policy learning, we relabel all the agent's past experience when its reward model changes. We additionally show that pre-training our agents with unsupervised exploration substantially increases the mileage of its queries. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods, including a variety of locomotion and robotic manipulation skills. We also show that our method is able to utilize real-time human feedback to effectively prevent reward exploitation and learn new behaviors that are difficult to specify with standard reward functions.
Introduction
Reinforcement Learning (RL) typically relies on a reward function to guide an agent toward a desired goal. However, designing these functions is a notorious "dark art." If the reward is too sparse, the agent learns nothing for a long time; if it is too dense or poorly designed, the agent might "exploit" the reward—finding unintended ways to accumulate points without actually performing the desired task.
Human-in-the-loop (HiL) RL offers a solution by allowing a human teacher to provide feedback, such as picking which of two behavior clips looks better. While effective, this process is expensive because human time is limited. Previous state-of-the-art methods were often "on-policy," meaning they discarded old data every time the reward model was updated, leading to extreme feedback inefficiency. PEBBLE addresses these challenges by making interactive RL more sample- and feedback-efficient.
The PEBBLE Framework
PEBBLE (unsupervised PrE-training and preference-Based learning via RElaBeLing Experience) introduces three key innovations to streamline the teaching process:
- Unsupervised Pre-training: Before any human feedback is requested, the agent explores the environment using intrinsic motivation. By maximizing "state entropy," the agent learns to visit a diverse range of states. This ensures that when human feedback begins, the agent is already demonstrating interesting behaviors rather than random movements.
- Off-policy Learning with Relabeling: PEBBLE utilizes Soft Actor-Critic (SAC), a sample-efficient off-policy algorithm. To handle the non-stationarity of a changing reward function, PEBBLE relabels the entire replay buffer with the latest reward model whenever it is updated. This allows the agent to re-interpret its entire history through its evolving understanding of the task.
- Entropy-based Query Selection: Instead of random selection, PEBBLE uses active learning to choose pairs of segments where the reward model is most uncertain (highest entropy), maximizing the information gained from each human interaction.
Experimental Evaluation
The authors evaluated PEBBLE across a variety of challenging continuous control benchmarks, including locomotion tasks from the DeepMind Control Suite and robotic manipulation tasks from Meta-world.
Beyond standard benchmarks, the framework was tested on "unscripted" behaviors, such as teaching a robot to wave its leg or a hopper to perform a backflip, using real-time human feedback.
Key Insights and Results
- Overcoming the "Boring Query" Problem: Unsupervised pre-training effectively bypasses the initial phase of training where agents often perform repetitive, uninformative actions, making early human queries much more productive.
- Efficiency and Performance: PEBBLE matched the performance of an "oracle" agent (using ground-truth rewards) with only 1,400 queries on complex locomotion tasks. It significantly outperformed previous standards like "Preference PPO" in both learning speed and final success rates.
- Complex Manipulation: The framework successfully solved Meta-world tasks that were previously considered too difficult for preference-based methods, demonstrating its scalability to high-dimensional manipulation.
- Human Feedback as a Regularizer: Human preferences naturally steer agents toward more "natural" movements, which helps prevent the "ugly" reward exploitation often seen with engineered reward functions.
Conclusion and Future Work
PEBBLE represents a significant step toward making human-in-the-loop RL practical for real-world applications. While highly effective, the approach does face limitations, such as the computational cost of relabeling large replay buffers and the potential for pre-training to fail in environments with high-entropy distractions (the "noisy TV" problem). Future research may focus on handling inconsistent human feedback and further optimizing the relabeling process for even larger datasets.
Summary updated