TL;DR
This paper introduces CLASP (Composable Learned Action Space Predictor), a model that learns to represent an agent's action space purely by watching unlabeled videos. By enforcing two key properties—minimality and composability—the model disentangles actions from the static environment. This allows for efficient planning and video prediction using orders of magnitude fewer labels than traditional supervised methods.
The Challenge of Learning by Observing
In traditional Reinforcement Learning (RL), agents learn by doing: they interact with the environment, receive rewards, and gradually figure out which actions lead to which outcomes. However, active interaction is often expensive, slow, or dangerous. Humans, by contrast, often learn by observing others before ever attempting a task themselves—think of an infant watching adults walk for months before taking their first step.
The technical challenge is learning a "clean" representation of actions from raw video. Most unsupervised video prediction models suffer from entanglement: the representation of a "move right" action might look different depending on whether the agent is in the center of the room or a corner. To be useful for planning, an action representation must be invariant to the scene's static content.
Methods: Minimality and Composability
The authors propose a framework called CLASP, built on top of a stochastic video prediction model. They introduce two primary constraints to force the model to learn a structured action space:
- Minimality (Information Bottleneck): Using the Information Bottleneck principle, the model is penalized for carrying too much information about the current frame in its latent variable ($z$). This forces $z$ to focus exclusively on the change between frames (the action) rather than the static state of the world.
- Composability: This is the core innovation. The authors introduce a loss term requiring that a sequence of actions can be mathematically composed into a single latent representation. This composed representation must predict the final outcome without needing to see the intermediate frames.
If the model can successfully predict the result of a "composed" action (e.g., "move right" then "move up"), it proves that the individual action representations are disentangled from specific starting states. Once the model is trained on unlabeled video, a tiny amount of labeled data is used to "ground" the latent space—mapping the learned $z$ vectors to actual control commands ($u$).
Key Insights and Results
The researchers evaluated CLASP on two main benchmarks: a Simulated Reacher (a 1-degree-of-freedom robotic arm with complex, shifting backgrounds) and BAIR Robot Pushing (a real-world dataset featuring a robotic arm moving objects on a table).
- Composition as a Disentangler: The "aha moment" of the paper is that composability forces disentanglement. If an action representation $z$ were "polluted" by information about the static background, you couldn't easily add two $z$ vectors together to predict a long-term result, because the background information would conflict.
- Extreme Data Efficiency: CLASP achieved high performance using only 100 to 1,000 labeled sequences. In contrast, supervised baselines required over 10,000 labels to reach similar accuracy. This represents a 10x to 100x improvement in label efficiency.
- Action-Conditioned Prediction & Planning: On the Reacher task, CLASP achieved an absolute prediction error of 2.9 degrees, nearly matching fully supervised methods (2.6 degrees) and vastly outperforming standard unsupervised baselines (22.6 degrees). When used for visual servoing (planning a path to a target image), CLASP reached the goal with an error of just 1.6 degrees.
- Robustness and Transfer: Because the model is forced to be "minimal," it naturally ignores visual distractors like changing backgrounds or the specific color of the robot arm. It can even learn an action space from a variety of different-looking robots and successfully apply that knowledge to a new, unseen robot.
Limitations and Future Work
While impressive, the paper notes a few areas for future growth:
- Action Complexity: The experiments focused on relatively low-dimensional action spaces (1-DOF or 2D pushing). It remains to be seen how well this scales to high-dimensional systems like humanoids or complex dexterous manipulation.
- Dynamic Backgrounds: The "minimality" constraint assumes the agent is the primary source of change. If the background itself is highly dynamic (e.g., a robot working in a crowded street), the model might struggle to separate agent actions from environmental noise.
- Scaling to "Wild" Video: A major future goal is applying this framework to uncurated video data, such as YouTube videos of humans performing tasks, to learn action spaces for robots without any manual data collection.
Summary updated