Abstract
Learning to control robots directly based on images is a primary challenge in robotics. However, many existing reinforcement learning approaches require iteratively obtaining millions of robot samples to learn a policy, which can take significant time. In this paper, we focus on learning a realistic world model capturing the dynamics of scene changes conditioned on robot actions. Our dreaming model can emulate samples equivalent to a sequence of images from the actual environment, technically by learning an action-conditioned future representation/scene regressor. This allows the agent to learn action policies (i.e., visuomotor policies) by interacting with the dreaming model rather than the real-world. We experimentally confirm that our dreaming model enables robot learning of policies that transfer to the real-world.
Introduction
Deep Reinforcement Learning (RL) has seen massive success in simulated environments, but the "reality gap" remains a formidable hurdle for physical robotics. Standard RL often requires millions of samples to converge. In the real world, a single robot trial can take minutes; running thousands of hours of trials is logistically difficult, expensive, and potentially damaging to hardware. This paper introduces a "dreaming" world model that allows robots to learn complex visuomotor policies entirely in a simulated latent space, significantly improving sample efficiency.
The Dreaming Model Architecture
The proposed system learns to "dream" realistic sequences of future states by training on a small set of random real-world actions. The architecture consists of three main components:
- Convolutional Variational Autoencoder (VAE): Unlike standard VAEs that compress images into a flat 1D vector, this model maintains a 3D tensor ($W \times H \times C$). This preserves spatial information critical for navigation.
- Action-Conditioned Future Regressor: This module takes the current latent state and a proposed robot action to predict the next latent state.
- Target Specific Decoders (TSD): A shared encoder works with multiple decoders specific to different objects. This forces the latent state to represent spatial positioning while decoders handle object appearance, facilitating zero-shot generalization.
Once trained on random data, the robot performs policy learning in its "dream." It samples a starting state and unrolls a trajectory by predicting future states based on its policy, using standard RL algorithms like Actor-Critic to optimize behavior without real-world risk.
Experimental Setup and Results
The model was evaluated using a Real-World Office Dataset (40,000 images from a TurtleBot 2) and a Gazebo Simulation for complex obstacle avoidance.
Key Insights
- Spatial Latent Spaces: Keeping the latent state as a 3D convolutional tensor significantly improves the prediction of movement consequences compared to 1D vectors.
- Dreaming vs. Imagining: By sampling from the VAE's latent prior, the model can explore a wider variety of starting conditions than those present in the initial training data.
- Decoupling "What" from "Where": The TSD approach allows the robot to learn general policies (e.g., "approach target") that work even for unseen objects.
Performance Metrics
The "Dreaming (Conv)" model achieved a prediction error ($D_1$ distance) of 0.71, outperforming standard baselines like Embed-to-Control (0.95). In real-world trials, the model achieved a 60-65% success rate on approach tasks and a 90-100% success rate on avoidance tasks. Notably, the system required as few as 5,000 random action samples to learn a functional model.
Conclusion
While the current model excels at navigation, the visual reconstructions lack fine-grained detail. Future work will focus on manipulation tasks requiring higher fidelity, handling sparse rewards without pre-trained goal detectors, and adapting to dynamic environments with moving obstacles or changing lighting conditions.
Summary updated