Data-efficient visuomotor policy training using reinforcement learning and generative models

Ali Ghadirzadeh, Petra Poklukar, Ville Kyrki, Danica Kragic, Mårten BjörkmanarXiv 2020 · arXiv

Abstract

Training robots to perform complex tasks from visual input typically requires massive amounts of data, particularly when rewards are sparse. This paper introduces a framework that leverages generative models, such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), to learn a "latent action space." By mapping high-dimensional motor trajectories to a low-dimensional latent variable, the search space for Reinforcement Learning (RL) is reduced by an order of magnitude. Furthermore, the authors propose new metrics—Disentangling Precision and Recall (DPR) and Latent Local Linearity (L3)—to predict robot performance based on generative model characteristics before physical training begins.

Introduction

Deep Reinforcement Learning has shown significant potential in visuomotor tasks, where robots must map raw pixel inputs directly to motor torques. However, RL is notoriously data-hungry. In scenarios with sparse terminal rewards, a robot may spend days performing random actions before achieving a single success. This paper addresses two primary challenges: reducing the number of real-world trials required for training and developing metrics to evaluate whether a generative model is suitable for a specific robotic task.

Methods

The proposed framework consists of a three-stage pipeline designed to simplify the RL exploration process:

  1. Unsupervised Learning: A generative model (VAE or InfoGAN) is trained on a dataset of valid action sequences (e.g., 15,750 joint velocity trajectories). The model learns to map a low-dimensional latent variable ($\alpha$) to a complete trajectory of motor commands ($\tau$).
  2. RL Policy Training: Instead of searching through every possible motor command at every millisecond, the RL agent selects a single latent variable $\alpha$ for the entire episode. This transforms a complex sequential decision-making problem into a simpler "contextual multi-armed bandit" problem.
  3. End-to-End Refinement: The vision and control modules are fine-tuned together using supervised learning to optimize performance.
The deep action-selection policy framework maps initial states to latent action variables, which are then converted into motor trajectories by a generative model.

To optimize the RL policy, the authors utilize an Expectation-Maximization (EM) approach with a trust-region term to ensure stable training. Additionally, two novel evaluation measures are introduced:

  • Disentangling Precision and Recall (DPR): Evaluates how well latent dimensions control specific physical task aspects and the coverage of the action distribution.
  • Latent Local Linearity (L3): Measures the smoothness of the transition between latent space points and resulting robot end-states.

Key Results and Insights

The framework was tested on a robotic picking task using a 7-degree-of-freedom ABB YuMi arm. Key findings include:

  • Efficiency Gains: The approach improves data efficiency by at least an order of magnitude compared to standard RL methods.
  • Recall is Critical: The most significant predictor of success is "Recall"—the generative model's ability to cover the entire distribution of valid actions. For VAEs, there was a near-perfect correlation (Pearson’s R = 0.969) between recall and final reward. If a model cannot generate a specific necessary movement, the RL agent can never learn to execute it.
  • Model Stability: VAEs generally proved more stable and easier to train for this task than GANs. For GANs, high disentanglement (DiR) was a stronger predictor of performance.
  • Safety: By sampling from a learned distribution of "valid" actions, exploration is inherently safer for physical hardware than taking completely random actions.

Conclusion

By bypassing the temporal credit assignment problem through generative modeling, this framework significantly reduces the data requirements for complex robotic tasks. While the current study focused on static picking tasks, the results suggest that "coverage" (recall) in the latent space is more vital than "smoothness" (linearity). Future work will explore more dynamic tasks and methods to reduce the dependency on pre-existing trajectory datasets for initial model training.

Summary updated

Abstract

We present a data-efficient framework for solving visuomotor sequential decision-making problems which exploits the combination of reinforcement learning (RL) and latent variable generative models. Our framework trains deep visuomotor policies by introducing an action latent variable such that the feed-forward policy search can be divided into three parts: (i) training a sub-policy that outputs a distribution over the action latent variable given a state of the system, (ii) unsupervised training of a generative model that outputs a sequence of motor actions conditioned on the latent action variable, and (iii) supervised training of the deep visuomotor policy in an end-to-end fashion. Our approach enables safe exploration and alleviates the data-inefficiency problem as it exploits prior knowledge about valid sequences of motor actions. Moreover, we provide a set of measures for evaluation of generative models such that we are able to predict the performance of the RL policy training prior to the actual training on a physical robot. We define two novel measures of disentanglement and local linearity for assessing the quality of latent representations, and complement them with existing measures for assessment of the learned distribution. We experimentally determine the characteristics of different generative models that have the most influence on performance of the final policy training on a robotic picking task.
BibTeX
@misc{ghadirzadeh2020data,
  author = {Ali Ghadirzadeh and Petra Poklukar and Ville Kyrki and Danica Kragic and Mårten Björkman},
  title = {{Data-efficient visuomotor policy training using reinforcement learning and generative models}},
  year = {2020},
  eprint = {2007.13134},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.13134},
}
Figure