Unsupervised Learning of Sensorimotor Affordances by Stochastic Future Prediction

Oleh Rybkin, Karl Pertsch, Andrew Jaegle, Konstantinos G. Derpanis, Kostas Daniilidis

Abstract

Intelligent agents can learn to represent the action spaces of other agents simply by observing them act. Such representations help agents quickly learn to predict the effects of their own actions on the environment and to plan complex action sequences. In this work, we address the problem of learning an agent's action space purely from visual observation. We use stochastic video prediction to learn a latent variable that captures the scene's dynamics while being minimally sensitive to the scene's static content. We introduce a loss term that encourages the network to capture the composability of visual sequences and show that it leads to representations that disentangle the structure of actions. We call the full model with composable action representations Composable Learned Action Space Predictor (CLASP). We show the applicability of our method to synthetic settings and its potential to capture action spaces in complex, realistic visual settings. When used in a semi-supervised setting, our learned representations perform comparably to existing fully supervised methods on tasks such as action-conditioned video prediction and planning in the learned action space, while requiring orders of magnitude fewer action labels. Project website: https://daniilidis-group.github.io/learned_action_spaces

Introduction

In traditional Reinforcement Learning (RL), agents typically learn by "doing"—exploring an environment until they discover which motor commands lead to desired outcomes. While effective, this process is data-hungry and often requires active access to physical hardware or high-fidelity simulators.

Humans, however, learn a "perceptual representation" of actions just by observing others. The challenge in AI is to learn a structured action space from raw pixels without knowing underlying motor commands or having access to labels. The Composable Learned Action Space Predictor (CLASP) addresses this by learning to understand an agent's possible actions simply by watching unlabeled videos.

Methods: Minimality and Composability

CLASP builds on stochastic video prediction but introduces two critical constraints to the latent representation ($z$):

  1. Minimality (Information Bottleneck): The model is forced to compress the information passed between frames. By penalizing the information content in the latent variable, the model learns to ignore static elements (like backgrounds) and focus exclusively on the change (the action).
  2. Composability: This is the core innovation. The authors argue that if an agent performs action A and then action B, the result should be the same as a single "composed" action AB. CLASP uses a loss term that forces the latent space to be additive. Because a composed representation lacks access to intermediate frames, the model is forced to learn an action representation that is independent of the starting state.
Figure 1: (a) The composability property where sequential actions are additive in the latent space. (b) Visualization of the resulting disentangled latent action space.

Key Insights and Results

The model was evaluated on the Simulated Reacher (a robotic arm with randomized backgrounds) and the BAIR Robot Pushing dataset.

Extreme Data Efficiency

CLASP demonstrates remarkable efficiency when grounded with a tiny amount of labeled data. On the Reacher task, it achieved high performance using only 100 labeled videos, whereas traditional supervised models required 10,000 videos to reach the same level of accuracy—a 100x improvement.

Disentanglement and Action Transplantation

Because the learned actions are disentangled from the scene, they can be "transplanted" between different environments. For example, a motion learned from a robot with long arms can be accurately applied to a robot with short arms in a different setting. In visual servoing tasks, CLASP significantly outperformed baselines, achieving a planning error of only 1.6 degrees compared to 14.1 degrees for standard stochastic video prediction.

Conclusion

CLASP proves that passive observation is a powerful pre-training step. By the time an agent is given actual control labels, it already understands the "physics" of the action space. This approach bridges the gap between human-like observation and robotic execution, potentially allowing robots to learn complex manipulations by watching "in-the-wild" videos of humans.

Summary updated

Abstract

Intelligent agents can learn to represent the action spaces of other agents simply by observing them act. Such representations help agents quickly learn to predict the effects of their own actions on the environment and to plan complex action sequences. In this work, we address the problem of learning an agent's action space purely from visual observation. We use stochastic video prediction to learn a latent variable that captures the scene's dynamics while being minimally sensitive to the scene's static content. We introduce a loss term that encourages the network to capture the composability of visual sequences and show that it leads to representations that disentangle the structure of actions. We call the full model with composable action representations Composable Learned Action Space Predictor (CLASP). We show the applicability of our method to synthetic settings and its potential to capture action spaces in complex, realistic visual settings. When used in a semi-supervised setting, our learned representations perform comparably to existing fully supervised methods on tasks such as action-conditioned video prediction and planning in the learned action space, while requiring orders of magnitude fewer action labels. Project website: https://daniilidis-group.github.io/learned_action_spaces
BibTeX
@inproceedings{rybkin2019unsupervised,
  author = {Oleh Rybkin and Karl Pertsch and Andrew Jaegle and Konstantinos G. Derpanis and Kostas Daniilidis},
  title = {{Unsupervised Learning of Sensorimotor Affordances by Stochastic Future Prediction}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=Hc_BggYkN5},
}
Figure