Learning what you can do before doing anything

Oleh Rybkin, Karl Pertsch, Konstantinos G. Derpanis, Kostas Daniilidis, Andrew Jaegle

TL;DR

This paper introduces CLASP (Composable Learned Action Space Predictor), a model that learns to represent an agent's action space purely by watching unlabeled videos. By enforcing two key properties—minimality and composability—the model disentangles actions from the static environment. This allows for efficient planning and video prediction using orders of magnitude fewer labels than traditional supervised methods.

The Challenge of Learning by Observing

In traditional Reinforcement Learning (RL), agents learn by doing: they interact with the environment, receive rewards, and gradually figure out which actions lead to which outcomes. However, active interaction is often expensive, slow, or dangerous. Humans, by contrast, often learn by observing others before ever attempting a task themselves—think of an infant watching adults walk for months before taking their first step.

The technical challenge is learning a "clean" representation of actions from raw video. Most unsupervised video prediction models suffer from entanglement: the representation of a "move right" action might look different depending on whether the agent is in the center of the room or a corner. To be useful for planning, an action representation must be invariant to the scene's static content.

Methods: Minimality and Composability

The authors propose a framework called CLASP, built on top of a stochastic video prediction model. They introduce two primary constraints to force the model to learn a structured action space:

  1. Minimality (Information Bottleneck): Using the Information Bottleneck principle, the model is penalized for carrying too much information about the current frame in its latent variable ($z$). This forces $z$ to focus exclusively on the change between frames (the action) rather than the static state of the world.
  2. Composability: This is the core innovation. The authors introduce a loss term requiring that a sequence of actions can be mathematically composed into a single latent representation. This composed representation must predict the final outcome without needing to see the intermediate frames.
Composability in CLASP: A sequence of actions (Right, then Up) can be composed into a single latent representation that predicts the final state.

If the model can successfully predict the result of a "composed" action (e.g., "move right" then "move up"), it proves that the individual action representations are disentangled from specific starting states. Once the model is trained on unlabeled video, a tiny amount of labeled data is used to "ground" the latent space—mapping the learned $z$ vectors to actual control commands ($u$).

Key Insights and Results

The researchers evaluated CLASP on two main benchmarks: a Simulated Reacher (a 1-degree-of-freedom robotic arm with complex, shifting backgrounds) and BAIR Robot Pushing (a real-world dataset featuring a robotic arm moving objects on a table).

  • Composition as a Disentangler: The "aha moment" of the paper is that composability forces disentanglement. If an action representation $z$ were "polluted" by information about the static background, you couldn't easily add two $z$ vectors together to predict a long-term result, because the background information would conflict.
  • Extreme Data Efficiency: CLASP achieved high performance using only 100 to 1,000 labeled sequences. In contrast, supervised baselines required over 10,000 labels to reach similar accuracy. This represents a 10x to 100x improvement in label efficiency.
  • Action-Conditioned Prediction & Planning: On the Reacher task, CLASP achieved an absolute prediction error of 2.9 degrees, nearly matching fully supervised methods (2.6 degrees) and vastly outperforming standard unsupervised baselines (22.6 degrees). When used for visual servoing (planning a path to a target image), CLASP reached the goal with an error of just 1.6 degrees.
  • Robustness and Transfer: Because the model is forced to be "minimal," it naturally ignores visual distractors like changing backgrounds or the specific color of the robot arm. It can even learn an action space from a variety of different-looking robots and successfully apply that knowledge to a new, unseen robot.

Limitations and Future Work

While impressive, the paper notes a few areas for future growth:

  • Action Complexity: The experiments focused on relatively low-dimensional action spaces (1-DOF or 2D pushing). It remains to be seen how well this scales to high-dimensional systems like humanoids or complex dexterous manipulation.
  • Dynamic Backgrounds: The "minimality" constraint assumes the agent is the primary source of change. If the background itself is highly dynamic (e.g., a robot working in a crowded street), the model might struggle to separate agent actions from environmental noise.
  • Scaling to "Wild" Video: A major future goal is applying this framework to uncurated video data, such as YouTube videos of humans performing tasks, to learn action spaces for robots without any manual data collection.

Summary updated

Abstract

Intelligent agents can learn to represent the action spaces of other agents simply by observing them act. Such representations help agents quickly learn to predict the effects of their own actions on the environment and to plan complex action sequences. In this work, we address the problem of learning an agent’s action space purely from visual observation. We use stochastic video prediction to learn a latent variable that captures the scene's dynamics while being minimally sensitive to the scene's static content. We introduce a loss term that encourages the network to capture the composability of visual sequences and show that it leads to representations that disentangle the structure of actions. We call the full model with composable action representations Composable Learned Action Space Predictor (CLASP). We show the applicability of our method to synthetic settings and its potential to capture action spaces in complex, realistic visual settings. When used in a semi-supervised setting, our learned representations perform comparably to existing fully supervised methods on tasks such as action-conditioned video prediction and planning in the learned action space, while requiring orders of magnitude fewer action labels. Project website: https://daniilidis-group.github.io/learned_action_spaces
BibTeX
@inproceedings{rybkin2019learning,
  author = {Oleh Rybkin and Karl Pertsch and Konstantinos G. Derpanis and Kostas Daniilidis and Andrew Jaegle},
  title = {{Learning what you can do before doing anything}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=Hc_BggYkN5},
}
Figure