Abstract
Training robots to perform complex tasks from visual input typically requires massive amounts of data, particularly when rewards are sparse. This paper introduces a framework that leverages generative models, such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), to learn a "latent action space." By mapping high-dimensional motor trajectories to a low-dimensional latent variable, the search space for Reinforcement Learning (RL) is reduced by an order of magnitude. Furthermore, the authors propose new metrics—Disentangling Precision and Recall (DPR) and Latent Local Linearity (L3)—to predict robot performance based on generative model characteristics before physical training begins.
Introduction
Deep Reinforcement Learning has shown significant potential in visuomotor tasks, where robots must map raw pixel inputs directly to motor torques. However, RL is notoriously data-hungry. In scenarios with sparse terminal rewards, a robot may spend days performing random actions before achieving a single success. This paper addresses two primary challenges: reducing the number of real-world trials required for training and developing metrics to evaluate whether a generative model is suitable for a specific robotic task.
Methods
The proposed framework consists of a three-stage pipeline designed to simplify the RL exploration process:
- Unsupervised Learning: A generative model (VAE or InfoGAN) is trained on a dataset of valid action sequences (e.g., 15,750 joint velocity trajectories). The model learns to map a low-dimensional latent variable ($\alpha$) to a complete trajectory of motor commands ($\tau$).
- RL Policy Training: Instead of searching through every possible motor command at every millisecond, the RL agent selects a single latent variable $\alpha$ for the entire episode. This transforms a complex sequential decision-making problem into a simpler "contextual multi-armed bandit" problem.
- End-to-End Refinement: The vision and control modules are fine-tuned together using supervised learning to optimize performance.
To optimize the RL policy, the authors utilize an Expectation-Maximization (EM) approach with a trust-region term to ensure stable training. Additionally, two novel evaluation measures are introduced:
- Disentangling Precision and Recall (DPR): Evaluates how well latent dimensions control specific physical task aspects and the coverage of the action distribution.
- Latent Local Linearity (L3): Measures the smoothness of the transition between latent space points and resulting robot end-states.
Key Results and Insights
The framework was tested on a robotic picking task using a 7-degree-of-freedom ABB YuMi arm. Key findings include:
- Efficiency Gains: The approach improves data efficiency by at least an order of magnitude compared to standard RL methods.
- Recall is Critical: The most significant predictor of success is "Recall"—the generative model's ability to cover the entire distribution of valid actions. For VAEs, there was a near-perfect correlation (Pearson’s R = 0.969) between recall and final reward. If a model cannot generate a specific necessary movement, the RL agent can never learn to execute it.
- Model Stability: VAEs generally proved more stable and easier to train for this task than GANs. For GANs, high disentanglement (DiR) was a stronger predictor of performance.
- Safety: By sampling from a learned distribution of "valid" actions, exploration is inherently safer for physical hardware than taking completely random actions.
Conclusion
By bypassing the temporal credit assignment problem through generative modeling, this framework significantly reduces the data requirements for complex robotic tasks. While the current study focused on static picking tasks, the results suggest that "coverage" (recall) in the latent space is more vital than "smoothness" (linearity). Future work will explore more dynamic tasks and methods to reduce the dependency on pre-existing trajectory datasets for initial model training.
Summary updated