Learning Robotic Control from Action-Free Video

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2411 papers528 conceptsfrozen 2026-10-04

The "data flywheel" that powered the revolution in Large Language Models (LLMs) has been notoriously difficult to replicate in robotics. While text is abundant on the internet, high-quality physical trajectories for robots—complete with motor torques and joint angles—are expensive, slow to collect, and hardware-specific Choi et al., 2025.

However, the internet contains a nearly infinite supply of "action-free" data: millions of hours of YouTube tutorials, cooking videos, and human-computer interactions. These videos show what happens in the world, but they lack the how—the underlying control signals (actions) that caused the visual transitions. Recent research focuses on bridging this "action-observation divide" to turn passive video into active robotic policy.

The Action Gap and the Inverse Dynamics Solution

The fundamental challenge is that standard imitation learning, such as Behavioral Cloning, requires pairs of states and actions $(s, a)$. In a video of a human opening a door, we see the state change, but we don't know the precise forces the human's muscles applied.

One dominant strategy is to treat action-labeling as an inverse dynamics problem. If an agent can learn a model that predicts "what action must have happened to get from frame A to frame B," it can self-label millions of hours of video. Qinqing Zheng and Mikael Henaff demonstrated in SS-ORL that agents can match the performance of fully labeled datasets using as little as 10% labeled data by using an inverse dynamics model to provide proxy actions for the rest Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories. Similarly, the Watch & Learn framework by Chan Hee Song and colleagues applies this to digital tasks, inferring clicks and keystrokes from raw pixels to create executable trajectories from web tutorials Watch and Learn: Learning to Use Computers from Online Videos.

Recovering Latent Action Spaces

What if we don't have any labeled actions at all? Researchers have begun developing methods to discover the "grammar" of actions directly from visual dynamics.

The Latent Action Policies (LAPO) framework, developed by Minqi Jiang and Dominik Schmidt, uses an unsupervised information bottleneck to force a model to "explain" state transitions through a quantized latent space Learning to Act without Actions. By observing how pixels change, the model recovers the underlying structure of the environment's action space without a single ground-truth label. This allows an agent to learn the "physics" of interaction—knowing that a certain latent "intent" results in a specific visual change—which can then be mapped to real robot motor commands with minimal fine-tuning.

Visual Dynamics as a Foundation

Instead of trying to guess the exact action, another branch of research focuses on Visual Dynamics Representations (VDR). The intuition here is that if a model can accurately predict the next frame in a video, it must have captured the causal structure of the world.

Bridging the Embodiment Gap

A subtle but critical problem in learning from human video is the "embodiment gap": a human hand looks and moves differently than a robotic gripper. If a robot tries to imitate a human too literally, it fails because it doesn't have fingers or skin.

Jiajun Wu and Chelsea Finn proposed a surprisingly simple solution: mask the hand Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations. By hiding the specific embodiment and focusing only on how the objects in the scene move, the model learns "agent-agnostic" dynamics. This allows the robot to focus on the effect of the action (e.g., the pot moving to the stove) rather than the specific visual appearance of the human hand. Shikhar Bahl and Abhinav Gupta took a similar approach with WHIRL, using an agent-agnostic objective to enable one-shot imitation of human videos in the wild Human-to-Robot Imitation in the Wild.

The Theoretical Limits of Video Pre-training

While the empirical results are promising, Dipendra Misra and John Langford have provided a theoretical cautionary note Towards Principled Representation Learning from Videos for Reinforcement Learning. They proved that "exogenous noise"—elements in the background that move independently of the agent (like a bird flying past a window)—makes learning from video exponentially harder than learning from action-labeled data. Without action labels, the model struggles to distinguish between changes it caused and changes that just happened. This suggests that while video is a powerful prior, the most efficient systems will likely remain those that combine massive passive observation with a small, high-quality "grounding" of real robotic interactions.

Papers in this version

Showing 60 of 411 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 370 papers
  2. Version 1 · 2026-10-04 · 411 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_state_only_imitation_learning_ca7f9d04,
  title = {state-only imitation learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-only-imitation-learning/rdr_ca7f9d041a60402db4a21999e8f0b2b4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_robotic_control_from_action_fre_ca7f9d04_v1,
  title = {Learning Robotic Control from Action-Free Video},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-robotic-control-from-action-free-video/rdr_ca7f9d041a60402db4a21999e8f0b2b4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}