Learning Real-World Robot Policies by Dreaming

A. J. Piergiovanni, Alan Wu, Michael S. Ryoo

Abstract

Learning to control robots directly based on images is a primary challenge in robotics. However, many existing reinforcement learning approaches require iteratively obtaining millions of robot samples to learn a policy, which can take significant time. In this paper, we focus on learning a realistic world model capturing the dynamics of scene changes conditioned on robot actions. Our dreaming model can emulate samples equivalent to a sequence of images from the actual environment, technically by learning an action-conditioned future representation/scene regressor. This allows the agent to learn action policies (i.e., visuomotor policies) by interacting with the dreaming model rather than the real-world. We experimentally confirm that our dreaming model enables robot learning of policies that transfer to the real-world.

Introduction

Deep Reinforcement Learning (RL) has seen massive success in simulated environments, but the "reality gap" remains a formidable hurdle for physical robotics. Standard RL often requires millions of samples to converge. In the real world, a single robot trial can take minutes; running thousands of hours of trials is logistically difficult, expensive, and potentially damaging to hardware. This paper introduces a "dreaming" world model that allows robots to learn complex visuomotor policies entirely in a simulated latent space, significantly improving sample efficiency.

The Dreaming Model Architecture

The proposed system learns to "dream" realistic sequences of future states by training on a small set of random real-world actions. The architecture consists of three main components:

  1. Convolutional Variational Autoencoder (VAE): Unlike standard VAEs that compress images into a flat 1D vector, this model maintains a 3D tensor ($W \times H \times C$). This preserves spatial information critical for navigation.
  2. Action-Conditioned Future Regressor: This module takes the current latent state and a proposed robot action to predict the next latent state.
  3. Target Specific Decoders (TSD): A shared encoder works with multiple decoders specific to different objects. This forces the latent state to represent spatial positioning while decoders handle object appearance, facilitating zero-shot generalization.
The dreaming model architecture (a) and its training objectives (b, c, d).

Once trained on random data, the robot performs policy learning in its "dream." It samples a starting state and unrolls a trajectory by predicting future states based on its policy, using standard RL algorithms like Actor-Critic to optimize behavior without real-world risk.

Experimental Setup and Results

The model was evaluated using a Real-World Office Dataset (40,000 images from a TurtleBot 2) and a Gazebo Simulation for complex obstacle avoidance.

Key Insights

  • Spatial Latent Spaces: Keeping the latent state as a 3D convolutional tensor significantly improves the prediction of movement consequences compared to 1D vectors.
  • Dreaming vs. Imagining: By sampling from the VAE's latent prior, the model can explore a wider variety of starting conditions than those present in the initial training data.
  • Decoupling "What" from "Where": The TSD approach allows the robot to learn general policies (e.g., "approach target") that work even for unseen objects.
A sequence of images representing the robot's visual input during a trajectory.

Performance Metrics

The "Dreaming (Conv)" model achieved a prediction error ($D_1$ distance) of 0.71, outperforming standard baselines like Embed-to-Control (0.95). In real-world trials, the model achieved a 60-65% success rate on approach tasks and a 90-100% success rate on avoidance tasks. Notably, the system required as few as 5,000 random action samples to learn a functional model.

Conclusion

While the current model excels at navigation, the visual reconstructions lack fine-grained detail. Future work will focus on manipulation tasks requiring higher fidelity, handling sparse rewards without pre-trained goal detectors, and adapting to dynamic environments with moving obstacles or changing lighting conditions.

Summary updated

Abstract

Learning to control robots directly based on images is a primary challenge in robotics. However, many existing reinforcement learning approaches require iteratively obtaining millions of robot samples to learn a policy, which can take significant time. In this paper, we focus on learning a realistic world model capturing the dynamics of scene changes conditioned on robot actions. Our dreaming model can emulate samples equivalent to a sequence of images from the actual environment, technically by learning an action-conditioned future representation/scene regressor. This allows the agent to learn action policies (i.e., visuomotor policies) by interacting with the dreaming model rather than the real-world. We experimentally confirm that our dreaming model enables robot learning of policies that transfer to the real-world.
BibTeX
@inproceedings{piergiovanni2019learning,
  author = {A. J. Piergiovanni and Alan Wu and Michael S. Ryoo},
  title = {{Learning Real-World Robot Policies by Dreaming}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=PZ0kZIHeNU},
}
Figure