One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning

Yuxin Jiang, Wenqiang Sun, Jiaqi Li, Yuchao Gu, Lvmin Zhang, Ivor Tsang, Mike Zheng ShouarXiv 2026 · arXiv

TL;DR

O$^2$LaF (One Prior, Two Dynamics) is a framework that repurposes a single pretrained video Diffusion Transformer (DiT) to learn "latent actions" from unlabeled videos. By using the same model as both an inverse dynamics encoder and a forward dynamics generator, it discovers a unified, transferable action space that enables zero-shot control across different embodiments, such as transferring motions from human demonstrations to robots.

What Problem Does This Paper Address?

While the internet is full of videos showing how the world changes, these videos rarely include the actual control signals (actions) that caused those changes. To build controllable world models or embodied agents, researchers often try to learn "latent actions"—mathematical representations of the "force" that transitions one frame to the next.

However, most existing methods learn these dynamics from scratch or only use pretrained models for half of the equation (e.g., using a frozen encoder for features). This paper asks: Can we leverage the rich temporal and physical priors already baked into large-scale video generators to learn both the "what happened" (inverse dynamics) and the "what will happen" (forward dynamics)?

Methods

The authors introduce O$^2$LaF, built on the principle of "one prior, two dynamics." They initialize two separate models from the same pretrained video DiT (specifically Wan2.2-5B) and adapt them using LoRA:

  1. Inverse Dynamics Model (IDM): This model acts as a transition encoder. It takes a video clip and uses learnable "action queries" to extract a compact, 32-dimensional latent action $z$ for each transition. Crucially, it uses an asymmetric attention mask so queries only "see" the relevant frames.
  2. Forward Dynamics Model (FDM): This is a causal, action-conditioned video generator. Given a starting frame and the latent actions $z$ from the IDM, it predicts the subsequent frames.
Overview of the O$^2$LaF framework showing how the IDM and FDM share a pretrained prior and interact through a latent action bottleneck.

To prevent the models from "cheating" (e.g., the FDM just copying pixels from a noisy target), the authors employ high-noise timestep sampling during training. By adding significant noise to the target frames, they force the FDM to rely on the latent action $z$ to reconstruct the scene, rather than relying on visual shortcuts.

Datasets

The models were trained and evaluated on several diverse datasets to test generalization:

  • Navigation: Trained on MiraData (large-scale cinematic/drone footage) and evaluated on MIND (navigation benchmark).
  • Manipulation: Trained on EgoDex (egocentric human hand-object interactions) and evaluated on In-lab GR-1 (real-world robot demonstrations).
  • Annotation: Evaluated on CameraBench for fine-grained camera motion recognition.

Key Insights and Ideas

  • Pretrained DiTs are "Action-Aware": Even without explicit action training, the cross-attention maps of a frozen video DiT naturally follow active hands and manipulated objects when prompted with "What is the action in the video?" (See Figure 3).
  • The "One Prior" Advantage: Initializing both the IDM and FDM from the same prior ensures they speak the same "visual language," leading to better alignment in the latent space than using mismatched backbones.
  • Noise as a Information Controller: In diffusion-based learning, noise isn't just for generation; it's a bottleneck. High noise in the FDM suppresses "target leakage," forcing the model to actually use the inferred actions to explain the future.
  • Probing vs. Control: A surprising finding was that models with the best "probing" accuracy (how well $z$ maps to real actions) don't always yield the best generative control. This highlights the need for end-to-end generative evaluation.

Key Results

O$^2$LaF sets a new state-of-the-art in several categories:

  • Few-Shot Probing: It significantly outperforms baselines like AdaWorld and DreamDojo in mapping latent actions to ground-truth controls (e.g., "turn right" or "arm +x") with very few labeled examples.
Linear probing results showing O$^2$LaF consistently outperforming prior methods in both navigation and robot manipulation tasks.
  • Zero-Shot Transfer: The model can take an action inferred from a human video and apply it to a robot, or take a 1st-person navigation action and apply it to a 3rd-person scene, without any target-domain fine-tuning.
  • Robot Manipulation: On the GR-1 robot benchmark, O$^2$LaF achieved an EEF (End-Effector) $R^2$ of 0.277, nearly doubling the performance of the previous best model, DreamDojo (0.149).

Limitations and Future Work

  • Computational Cost: Because it relies on large Video DiTs, inference (generation) is computationally expensive compared to smaller, non-diffusion world models.
  • Interpretability: While the latent actions are effective, the 32-D vector space remains a "black box." Making these actions more compositional or human-readable is a key next step.
  • Planning: Currently, O$^2$LaF is optimized for prediction. Bridging these visual actions to efficient planning and closed-loop control in a real-time robotics stack remains an open challenge.

Summary updated

Abstract

Videos reveal how the world changes, but rarely the actions that cause those changes. Latent action learning recovers such supervision from visual transitions, yet often learns the underlying dynamics largely from scratch. We ask whether a pretrained video generator can instead provide the prior for both inverse and forward dynamics. We introduce O$^{2}$LaF, built on a simple principle: one prior, two dynamics. We initialize both models from the same pretrained video DiT and adapt them to their distinct roles: a feed-forward transition encoder with learnable action queries for inverse dynamics, and a causal, action-conditioned video generator for forward dynamics. By controlling information access and optimizing both dynamics end-to-end through a compact bottleneck, O$^{2}$LaF learns transferable actions from unlabeled videos. Across navigation and robotic manipulation, O$^{2}$LaF sets a new state of the art in few-shot action probing and enables stronger zero-shot action transfer across visual contexts and embodiments. Beyond generative video priors, we further extend our framework to predictive representation priors such as V-JEPA2, achieving competitive performance. Our findings suggest a broader paradigm for latent-action learning: repurposing pretrained video models to discover transferable actions and predict their consequences.
BibTeX
@misc{jiang2026one,
  author = {Yuxin Jiang and Wenqiang Sun and Jiaqi Li and Yuchao Gu and Lvmin Zhang and Ivor Tsang and Mike Zheng Shou},
  title = {{One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning}},
  year = {2026},
  eprint = {2611.26617},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26617},
}
Figure