Demonstration-Bootstrapped Autonomous Practicing via Multi-Task Reinforcement Learning

Abhishek Gupta, Corey Lynch, Brandon Kinman, Garrett Peake, Sergey Levine, Karol Hausman

TL;DR

DBAP is a reinforcement learning (RL) framework that uses a small amount of human "play" data to build a task graph, allowing robots to practice complex multi-stage tasks autonomously. By treating different tasks as "resets" for one another, the system reduces human intervention by an order of magnitude while mastering long-horizon behaviors in real-world environments.

What Problem Does This Paper Address?

Reinforcement learning has shown great promise for robotics, but it faces a massive "reality gap" regarding human effort. Most RL algorithms assume episodic resets—the idea that after every attempt, the environment magically returns to its starting state. In a real kitchen, if a robot opens a cabinet to practice, a human must eventually walk over and close it so the robot can try again.

Furthermore, long-horizon tasks (e.g., "open the cabinet, then turn on the stove, then slide the door") are notoriously difficult for RL agents to learn from scratch because the odds of stumbling upon the correct sequence through random exploration are near zero. The authors aim to create a system that can learn these complex sequences with minimal human supervision.

Methods

The authors introduce Demonstration-Bootstrapped Autonomous Practicing (DBAP), which consists of three main phases:

  1. Data Collection: A human provides roughly 2.5 hours of "play-style" demonstrations—unstructured, continuous interactions where they perform various tasks (opening/closing things) in a sequence.
  2. Bootstrapping:
    • Low-Level Policy: The system uses AWAC (Advantage Weighted Actor-Critic), an offline RL algorithm, to initialize a goal-conditioned policy from the human data.
  3. High-Level Task Graph: The system builds a directed graph $\mathcal{G}$ where nodes are discrete goal states (e.g., "cabinet open, slider closed") and edges represent transitions observed in the data.
  4. Autonomous Practicing: During training, the robot doesn't just try the same task repeatedly. It uses graph search (Dijkstra’s algorithm) to select goals that maximize the entropy (uniformity) of its experience. If it needs to practice opening a door that is already open, the graph tells it to first perform the "close door" task. This allows the robot to "reset" itself.
  5. Test-Time Execution: To solve a long-horizon goal, the high-level graph search identifies a sequence of subgoals, and the low-level policy executes them one by one.

Datasets

  • Real-World Kitchen: A 7-DoF Franka Emika robotic arm in a physical kitchen setup with a sliding door, a cabinet, and a burner knob. The authors collected ~500 "play-style" demonstrations.
  • Simulation: A MuJoCo-based kitchen environment used for systematic comparisons against baselines, focusing on combinations of cabinet and slider configurations.

Key Insights and Ideas

  • Resets are just other tasks: Instead of training a specific "reset controller," DBAP recognizes that in a multi-task environment, many tasks are naturally the inverse of others. Practicing "closing" is a reset for "opening."
  • Graphs beat Neural Networks for Sequencing: The authors found that high-level policies trained via Behavior Cloning (BC) often suffer from "cycles" and sub-optimal paths. Using a simple graph search over discrete states is more robust and guarantees the shortest path to a goal.
  • Maximum Entropy for Coverage: By picking goals that lead to a uniform distribution of states, the robot ensures it doesn't just get stuck practicing the "easy" tasks, leading to better overall mastery of the environment.

Key Results

  • High Success Rate: In the real world, DBAP achieved a 95% success rate on long-horizon tasks, significantly outperforming pure offline RL (83%) and imitation learning (62%).
  • Minimal Human Effort: The system required a human reset only once per hour (every 50 trajectories). This is an order of magnitude less intervention than standard RL approaches.
  • Efficiency: DBAP found significantly shorter paths to goals than baselines. In simulation, it reached a 90% success rate while other methods struggled to sequence tasks effectively.

Limitations and Future Work

  • State Discretization: Currently, the system relies on thresholding continuous states into discrete "tasks" (e.g., a door is either "open" or "closed"). The authors suggest using clustering or representation learning to automate this for higher-dimensional problems.
  • Irreversible States: The algorithm assumes the environment is "reset-free," meaning the robot can't end up in a state it can't recover from (like knocking a kettle off a table). Handling truly irreversible actions remains an open challenge.
  • Vision-Based Learning: The current implementation uses state estimation (object positions). Moving toward learning directly from raw pixels would make the system more generalizable to unstructured homes.

Summary updated

Abstract

Reinforcement learning systems have the potential to enable continuous improvement in unstructured environments, leveraging data collected autonomously. However, in practice these systems require significant amounts of instrumentation or human intervention to learn in the real world. In this work, we propose a system for reinforcement learning that leverages multi-task reinforcement learning bootstrapped with prior data to enable continuous autonomous practicing, minimizing the number of resets needed while being able to learn temporally extended behaviors. We show how appropriately provided prior data can help bootstrap both low-level multi-task policies and strategies for sequencing these tasks one after another to enable learning with minimal resets. This mechanism enables our robotic system to practice with minimal human intervention at training time while being able to solve long horizon tasks at test time. We show the efficacy of the proposed system on a challenging kitchen manipulation task both in simulation and in the real world, demonstrating the ability to practice autonomously in order to solve temporally extended problems.
BibTeX
@inproceedings{gupta2023demonstration,
  author = {Abhishek Gupta and Corey Lynch and Brandon Kinman and Garrett Peake and Sergey Levine and Karol Hausman},
  title = {{Demonstration-Bootstrapped Autonomous Practicing via Multi-Task Reinforcement Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10161447},
}
Figure