PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training

Kimin Lee, Laura M. Smith, Pieter Abbeel

Introduction

Conveying complex objectives to reinforcement learning (RL) agents is a notoriously difficult challenge. Traditionally, RL relies on a mathematical "reward function" to tell the agent when it is succeeding. However, designing these functions often requires expensive sensors (like thermal cameras or accelerometers) to provide dense feedback. Furthermore, agents are prone to "reward exploitation"—finding unintended, lazy shortcuts to maximize rewards, such as a robot learning to hop on one leg to maximize forward velocity instead of walking naturally.

Human-in-the-loop (HiL) RL addresses this by allowing humans to provide direct feedback, often by simply indicating a preference between two video clips of the agent's behavior. The primary bottleneck? Human time is expensive. Previous HiL methods were largely "on-policy," meaning they discarded old experience data every time the reward model was updated. This made them incredibly slow and hungry for human feedback.

PEBBLE is a framework that makes HiL reinforcement learning significantly more practical. By combining unsupervised exploration with off-policy learning, PEBBLE achieves state-of-the-art performance on complex robotic tasks with a fraction of the human effort previously required.

The PEBBLE Framework

PEBBLE (unsupervised PrE-training and preference-Based learning via relaBeLing Experience) introduces three key innovations to solve the efficiency bottleneck of previous interactive RL systems.

PEBBLE Framework Overview
  1. Unsupervised Pre-training: Random exploration is a terrible way to start a human-in-the-loop process, often referred to as the "cold start" problem. Before asking a human for any feedback, the PEBBLE agent explores its environment using "intrinsic motivation" (specifically, maximizing state entropy). This ensures that when the agent finally asks for feedback, it presents the human with a diverse range of coherent behaviors rather than just twitching randomly.
  2. Off-policy Learning with Relabeling: The authors utilize Soft Actor-Critic (SAC), an off-policy algorithm that reuses old data stored in a "replay buffer." Crucially, human feedback changes the "ground truth" of the task mid-stream. Whenever the human provides new feedback and the reward model is updated, PEBBLE relabels every single transition in the replay buffer with the new reward. This allows the agent to learn from its entire history through the lens of the most up-to-date human preferences.
  3. Entropy-based Query Selection: Instead of picking random clips for the human to judge, PEBBLE selects pairs of behaviors where the reward model is most "uncertain" (highest entropy). This maximizes the information gained from every single human interaction.

Datasets and Environments

The authors evaluated PEBBLE across two major simulation benchmarks to test its versatility in both locomotion and manipulation:

  • DeepMind Control Suite (DMControl): Complex locomotion tasks including Cheetah-run, Walker-walk, and Quadruped-walk.
  • Meta-world: A benchmark for robotic manipulation involving tasks like Drawer Open, Door Open, Button Press, and Sweep Into.
Quadruped Environment

Additionally, they tested the system's ability to learn novel, non-standard tasks from actual human teachers, such as making a robot "wave" its leg or perform a backflip.

Key Results and Insights

PEBBLE demonstrates that data recycling is essential for interactive RL. By relabeling the replay buffer, modern off-policy algorithms can finally be used effectively in interactive settings.

  • State-of-the-Art Feedback Efficiency: On the complex Quadruped-walk task, PEBBLE matched the performance of an "Oracle" agent (which had access to a perfect, mathematically engineered reward) using only 1,400 human queries.
  • Outperforming Baselines: PEBBLE significantly outperformed "Preference PPO," the previous state-of-the-art. In many benchmark cases, the baseline failed to learn the task entirely within the same feedback budget.
  • Fixing "Hacks" and Qualitative Success: The paper shows that human preferences can lead to better, more natural behaviors than engineered rewards. The authors successfully trained a "Hopper" robot to do a backflip and a "Cartpole" to swing a pole in a specific way using only 50 human queries—tasks that are nearly impossible to define with simple distance-based reward functions.

Limitations and Future Work

While PEBBLE represents a massive leap forward for interactive RL, a few challenges remain:

  • Computational Overhead: Relabeling the entire replay buffer every time the reward model updates becomes computationally expensive as the agent collects millions of steps of experience.
  • Human Consistency: The underlying mathematical model assumes humans are relatively consistent in their preferences. It does not explicitly account for human fatigue, mistakes, or "drifting" preferences over a long training session.
  • Scaling to the Real World: While tested in high-fidelity simulations, moving this framework to physical robots will require handling the complexities of real-world resets and ensuring strict safety constraints during the unsupervised exploration phase.

Summary updated

Abstract

Conveying complex objectives to reinforcement learning (RL) agents can often be difficult, involving meticulous design of reward functions that are sufficiently informative yet easy enough to provi...
BibTeX
@inproceedings{lee2021pebble,
  author = {Kimin Lee and Laura M. Smith and Pieter Abbeel},
  title = {{PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=SZbwBCBMzq},
}
Figure