Traditional robot learning faces a severe data bottleneck. Training an agent to perform real-world manipulation tasks typically requires either thousands of hours of expensive, manually collected teleoperation data or highly engineered physics simulations that struggle to replicate the messy complexity of the real world.
To bypass this bottleneck, researchers are turning to Learning from Videos (LfV). The core intuition is simple: the internet contains a virtually infinite supply of video data showing humans interacting with the physical world. If a robot can leverage these "actionless" videos, it can acquire a foundational understanding of physics, object permanence, and affordances before ever touching a physical object.
As surveyed by Robert McCarthy et al. in Towards Generalist Robot Learning from Internet Video: A Survey, this paradigm shifts the burden of learning world dynamics from scarce robotic data to web-scale visual data.
Pixels as a Universal Interface
The primary challenge of learning from internet video is the embodiment gap. A YouTube video of a human slicing bread does not contain motor torques, joint angles, or low-level control signals. Furthermore, a human hand has a completely different morphology than a parallel-jaw robot gripper.
To overcome this, researchers treat pixels as a universal interface. Sherry Yang et al. argue in Position: Video as the New Language for Real-World Decision Making that video generation can serve as a unified medium for representation and planning. Instead of mapping an image directly to a robot action, a high-level planner can simply "dream" the future visual states of the environment.
[Current Image] + [Text Prompt] ---> (Video Diffusion Model) ---> [Generated Visual Plan (Subgoals)]
|
v
[Robot Action] <------------------- (Inverse Dynamics) <-------------------+
This approach is crystallized in UniPi, a framework developed by Yilun Du et al. in Learning Universal Policies via Text-Guided Video Generation. UniPi uses a text-to-video diffusion model to generate a sequence of frames showing the successful completion of a task. Because the video model is trained on massive, non-robotic datasets, it can generate physically plausible visual plans for novel objects and environments. A separate, lightweight inverse dynamics model then looks at these generated frames and calculates the specific physical actions the robot must take to match those visual transitions.
Similarly, Kevin Black et al. introduce SuSIE (Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models), which repurposes pre-trained image-editing models to synthesize visual subgoals. By framing robot planning as a series of image-to-image transformations, the system can manipulate objects it has never encountered before by borrowing semantic and physical knowledge from human-centric datasets.
Video Models as Learned Physics Simulators
Why does video generation transfer so effectively to robotic control? To generate a realistic sequence of future frames, a model must implicitly learn the laws of physics. It must understand that an object will fall if dropped, that cloth deforms when pulled, and that solid objects cannot pass through one another.
Homanga Bharadhwaj argues in Position: Scaling Simulation is Neither Necessary Nor Sufficient for In-the-Wild Robot Manipulation that manually engineering simulators to capture these complex real-world interactions is fundamentally unscalable. Instead, large-scale passive video datasets provide a more representative and scalable source for learning generalizable behavior.
This emergent physical intuition is demonstrated by Genie, an 11-billion parameter world model trained by Jake Bruce et al. (Genie: Generative Interactive Environments). Genie learns to generate interactive, playable virtual environments entirely from unlabeled internet videos. Without any ground-truth action labels, the model discovers a discrete set of latent actions, capturing intricate physical behaviors like soft-object deformation purely by observing visual transitions.
By leveraging this visual foresight, robots can plan through complex, non-rigid interactions that defy traditional geometric modeling. For instance, Boyang Wang demonstrates in This&That: Language-Gesture Controlled Video Generation for Robot Planning that future pixel prediction allows a robot to model complex tasks like cloth folding or object sliding without requiring explicit 3D geometric constraints. As noted by Jianke Zhang et al. in UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent, predicting subsequent frames forces the model to internalize 3D spatial relationships, directly translating to higher precision in downstream manipulation tasks.
Extracting Morphology-Invariant Priors
Beyond full-frame video generation, passive human videos can be mined for structural motion priors that are invariant to the specific agent performing the task.
Visual Representation Pretraining
One approach is to train a robust visual encoder that maps images into a feature space optimized for manipulation. Tete Xiao demonstrates in Real-World Robot Learning with Masked Visual Pre-training (MVP) that self-supervised pre-training using Masked Autoencoders (MAE) on massive egocentric human video datasets creates a universal visual prior. This prior generalizes across vastly different robot hardware configurations—such as standard arms and multi-fingered hands—because the underlying video data captures fundamental human-object interactions.
Similarly, Suraj Nair introduces R3M (R3M: A Universal Visual Representation for Robot Manipulation), showing that combining temporal consistency and language alignment on human videos yields a visual backbone that significantly improves the sample efficiency of real-world robotic controllers.
Affordance and Trajectory Modeling
Rather than learning passive features, models can explicitly track how objects move across time. Chuan Wen et al. propose ATM (Any-point Trajectory Modeling for Policy Learning), which pre-trains a transformer to predict the future 2D trajectories of arbitrary points within a video frame. By focusing on point-tracks, ATM extracts a structured motion prior from action-free videos that drastically reduces the amount of teleoperated data needed for behavioral cloning.
[Human Video Frame] ---> Track Hand-Object Trajectory ---> Morphology-Invariant Affordance Map
|
v
[Robot Workspace] ---> Align Gripper to Trajectory ---> Low-Data Skill Execution
To ground these trajectories in physical execution, Shikhar Bahl et al. develop SWIM (Structured World Models from Human Videos). SWIM extracts hand-object interaction trajectories from egocentric human videos to serve as proxy actions. By pre-training a world model on where humans grasp and move objects, a physical robot can inherit these visual affordances, allowing it to master new manipulation tasks with less than 30 minutes of real-world interaction data.
Grounding and Closed-Loop Execution
While generating a visual plan provides a powerful high-level blueprint, executing it in the real world introduces a final challenge: grounding. As pointed out by Qingwen Bu et al. in Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation, open-loop visual planners often suffer from compounding errors; a slight physical deviation can cause the robot to drift entirely from its "dreamed" trajectory.
To ground these visual models without relying on expensive human annotations, Yunhao Luo et al. propose a framework in Grounding Video Models to Actions through Goal Conditioned Exploration. The robot uses an internet-scale video model to generate visual subgoals, and then employs self-exploration and action chunking to discover the exact motor commands required to reach those subgoals. This allows the agent to ground the video model's abstract physical knowledge into its own specific embodiment without requiring any human-provided action labels or reward functions.
Finally, rather than separating prediction and control into two distinct steps, Yanjiang Guo introduces PAD (Prediction with Action: Visual Policy Learning via Joint Denoising Process). PAD unifies robot action generation and future image prediction into a single, joint denoising process using a Diffusion Transformer. By forcing the model to simultaneously predict how the world will look and how the robot should move, the system tightly couples visual foresight with physical execution, enabling robust, closed-loop generalization in unfamiliar environments.