To train an autonomous agent to perform complex tasks, traditional machine learning pipelines require paired data: a sequence of what the agent saw (states) matched precisely with what the agent did (motor commands or actions). However, collecting high-quality, action-labeled datasets for every robot embodiment or digital task is notoriously slow and expensive.
Meanwhile, an ocean of passive video data already exists. The internet is filled with millions of hours of humans cooking, playing video games, and navigating software interfaces. As noted by McCarthy et al. in their survey on Learning from Videos (LfV), these massive video repositories are currently underutilized because they lack low-level control signals (Wen et al.).
The core insight of latent action inference is that we do not need explicit action labels to learn how the world works. If a video frame changes from an egg being whole to an egg being cracked, an underlying action must have caused that transition. By treating actions as hidden, latent variables inferred directly from visual changes, models can transform passive video into rich, interactive training data.
The Mechanics of Inverse Dynamics
How do we extract an action from raw pixels? The primary mechanism is an inverse dynamics model. While a forward dynamics model predicts the next frame given the current frame and an action, an inverse dynamics model works backward: it takes two consecutive frames ($I_t$ and $I_{t+1}$) and infers the latent representation ($z_t$) that bridges them.
[ Frame t ] -----\
+---> [ Inverse Dynamics Model ] ---> [ Latent Action z_t ]
[ Frame t+1 ] ----/
This approach allows models to reconstruct control schemes across vastly different environments:
- Video Games: When ground-truth telemetry is missing, Pearce et al. demonstrated that an inverse dynamics model can successfully reconstruct discrete keyboard and mouse inputs from raw gameplay footage, enabling behavioral cloning at scale.
- Digital Interfaces: In the computer-use domain, Lu et al. developed an inverse dynamics module for VideoAgentTrek that automatically extracts precise click coordinates and keystroke strings from unlabeled screen recordings, turning passive tutorials into executable trajectories.
To prevent the model from simply memorizing pixel noise, researchers often pass the inferred transitions through a discrete bottleneck using Vector Quantization (VQ). As shown by Pan et al. and in the DeFI framework by Wenyao Zhang et al., forcing continuous visual changes into a discrete set of categorical codes acts as a powerful regularizer. It compels the network to group structurally similar physical movements (such as "swiping left" or "picking up an object") into the same discrete behavioral primitive.
Disentangling Motion from Context
A major challenge when learning from raw web video is that frames are cluttered with irrelevant information—changing backgrounds, lighting shifts, and visual distractors. If an inverse model tries to capture every single pixel change, the latent action space becomes polluted with environmental noise.
To solve this, models rely on an information bottleneck to separate motion dynamics from static context. By restricting the capacity of the latent action channel, the network is forced to discard everything that remains constant or irrelevant between frames, capturing only the causal delta.
For example, Gao et al. developed AdaWorld, a framework that uses this exact disentanglement to transfer physical behaviors across completely different visual domains without retraining. Because the latent action space only encodes the physical manipulation itself, a robot can learn a skill in one environment and seamlessly execute it in another.
Alternatively, this transition can be structured geometrically. Hayashi et al. proposed modeling actions as continuous transformations using Lie group theory, calculating the precise algebraic parameters needed to shift one state representation to the next. By optimizing purely for the reconstruction of the next state based on these transformations (Davtyan et al.), the model naturally ignores background distractors—a property that Wang et al. showed is crucial for keeping world models focused on task-relevant dynamics in cluttered scenes.
Building Playable Worlds and Policies
Once a model can reliably discover and encode latent actions from unlabeled videos, it unlocks several powerful downstream applications.
1. Generative Interactive Environments
By pairing an inverse action-discovery model with a forward frame-prediction model, researchers can build fully controllable simulations out of static video datasets. A prominent example is Genie, developed by Bruce et al., which learned consistent control schemes (like running, jumping, and navigating obstacles) purely from internet videos. This allows users to interact with and "play" generated environments frame-by-frame, effectively turning passive video into an interactive sandbox (Menapace et al.).
2. Cross-Embodiment Learning
Human bodies, robot arms, and digital avatars look completely different, but they obey shared physical or structural rules when interacting with the world. By mapping video transitions into an embodiment-agnostic latent space, frameworks like POI (Xie et al.) and UniSkill (Cho et al.) allow robots to extract functional interaction knowledge directly from human videos. To scale this across diverse hardware, Huang et al. developed LAC-WM to learn a unified "universal language" of actions that generalizes across entirely different robotic embodiments.
3. Mathematical Grounding and Policy Alignment
Can we trust that these unsupervised latent actions actually correspond to real physical actions? Lachapelle provided a formal proof in On the Identifiability of Latent Action Policies showing that an entropy-regularized objective can uniquely identify latent actions that are deterministic, state-independent, and fully representative of true underlying controls.
Once this latent space is established, an agent only needs a tiny fraction of labeled data to map the discovered latent codes to real-world motor commands. Alternatively, frameworks like SWIRL (Qiu et al.) use self-improving reinforcement learning loops where the forward world model and the inverse action model iteratively refine each other, bypassing human annotation entirely to master complex environments.
Key Foundational Work
Action-Agnostic Human Pose Forecasting (2018) — contributed 2 concept elements to this direction.
Learning Latent Super-Events to Detect Multiple Activities in Videos (2018) — contributed 1 concept elements to this direction.
Watch-Bot: Unsupervised Learning for Reminding Humans of Forgotten Actions (2015) — contributed 1 concept elements to this direction.