Abstract
Intelligent agents can learn to represent the action spaces of other agents simply by observing them act. Such representations help agents quickly learn to predict the effects of their own actions on the environment and to plan complex action sequences. In this work, we address the problem of learning an agent's action space purely from visual observation. We use stochastic video prediction to learn a latent variable that captures the scene's dynamics while being minimally sensitive to the scene's static content. We introduce a loss term that encourages the network to capture the composability of visual sequences and show that it leads to representations that disentangle the structure of actions. We call the full model with composable action representations Composable Learned Action Space Predictor (CLASP). We show the applicability of our method to synthetic settings and its potential to capture action spaces in complex, realistic visual settings. When used in a semi-supervised setting, our learned representations perform comparably to existing fully supervised methods on tasks such as action-conditioned video prediction and planning in the learned action space, while requiring orders of magnitude fewer action labels.
Introduction
In traditional Reinforcement Learning (RL), an agent learns by "doing"—interacting with an environment and receiving feedback. However, this requires active exploration and often millions of labeled data points (e.g., knowing exactly which motor command produced which movement).
Humans don't always learn this way. An infant watches adults walk for months before making their first attempt. This "passive" observation helps them understand the structure of movement before they ever touch the ground. The challenge in AI is: How can a model learn what actions are possible just by watching video, without any labels?
The primary technical hurdle is "disentanglement." In a video, the pixels represent both the static background (the "state") and the moving agent (the "action"). Standard models often get these confused, failing to realize that "moving left" is the same action regardless of whether the agent is in front of a blue wall or a red one.
Methods: The CLASP Architecture
The authors introduce CLASP, which builds on stochastic video prediction but adds two critical constraints to the latent variable $z$ (the mathematical representation of an action):
- Minimality (Information Bottleneck): The model is forced to compress the information in $z$. By penalizing the amount of information $z$ carries about the current frame, the model is incentivized to ignore the static background and focus only on the change between frames.
- Composability: This is the core contribution. The authors introduce a loss term requiring that a sequence of actions (e.g., $z_1$ then $z_2$) can be represented by a single "composed" latent variable $\nu$. For this to work, the representation must be independent of the starting state. If the representation were "tangled" with the background, you couldn't easily add two actions together without knowing exactly where you started.
Once this abstract action space is learned purely from video, the authors "ground" it using a tiny amount of labeled data to create a mapping between the latent $z$ and real-world motor commands $u$.
Experimental Setup
The researchers evaluated CLASP on two main benchmarks:
- Simulated Reacher: A 1-degree-of-freedom robot arm in a controlled environment. They created harder versions of this by adding random CIFAR-10 images as backgrounds and varying the arm's length and width to test robustness.
- BAIR Robot Pushing: A popular real-world dataset featuring a robotic arm moving objects on a table, which is significantly more complex and "noisy" than the simulation.
Key Insights
- Composability Implies Disentanglement: Forcing actions to be "additive" (composable) naturally pushes the model to separate the "action" from the "scene." If the model knows that "Step A + Step B = Long Jump," it can't afford to let the color of the floor leak into the definition of "Step A."
- Action Transplantation: Because the learned actions are disentangled, you can perform "transplantation." You can take the latent actions from a "donor" video (e.g., a thin arm moving in a circle) and apply them to a "recipient" (a thick arm on a different background). The recipient will perform the exact same motion perfectly.
- Learning from Others: The model can learn an action space by watching one type of agent and then apply that knowledge to control a different-looking agent, provided they share the same underlying physics.
Results
- Data Efficiency: CLASP achieves performance comparable to fully supervised models while using 100x to 1000x fewer action labels. In the Reacher task, CLASP reached a goal with an error of only 1.6 degrees using just 100 labeled videos, while supervised baselines were still struggling at over 30 degrees of error.
- Video Prediction: On the BAIR dataset, CLASP's action-conditioned predictions were significantly sharper and more accurate than standard unsupervised baselines, reducing the error gap between unsupervised and supervised methods by 30%.
- Robustness: The model successfully ignored complex, changing backgrounds (CIFAR-10) and variations in robot appearance, proving that the learned latent space truly captured the "physics" of the action rather than just pixel correlations.
Conclusion and Future Work
CLASP demonstrates that structured action spaces can be learned from passive observation. While the experiments focused on relatively low-dimensional action spaces (1-DOF reacher and 2D pushing), future work will explore scaling to high-dimensional "humanoid" robots. Additionally, while CLASP needs very few labels, it still requires some labeled data to map the abstract $z$ to real motor controls. Future work might explore "zero-shot" grounding through self-exploration.
Summary updated