Introduction
In recent years, deep learning has achieved transformative breakthroughs in fields like natural language processing and video generation by scaling models on massive, diverse internet datasets. However, robotics has yet to experience a similar "scaling moment." The primary constraint remains a severe data bottleneck: while text and images are abundant, high-quality robotic data—typically gathered through labor-intensive teleoperation or slow, specialized simulations—is scarce and expensive to produce.
Learning from Videos (LfV) has emerged as a promising paradigm to bridge this gap. By leveraging the billions of hours of human activity and physical interactions available on platforms like YouTube and TikTok, researchers aim to provide robots with a foundational understanding of the world. This survey systematically examines how internet video can be used to teach robots about physical dynamics, behaviors, and complex tasks, moving the field closer to the goal of general-purpose embodied intelligence.
The Core Challenges of LfV
While internet video is a goldmine of information, it cannot be directly ingested by robotic controllers due to two fundamental discrepancies:
- Distribution Shift (The Embodiment Gap): There is a significant visual and physical mismatch between video data and robotic execution. A human hand peeling an orange in a dimly lit kitchen looks and moves differently than a metallic robotic gripper in a controlled laboratory environment. This shift makes it difficult for a robot to map what it sees to its own physical form.
- Missing Action Labels: This is perhaps the most significant technical hurdle. Unlike robot demonstration data, which records specific motor commands (joint torques or velocities), internet videos only show the results of actions. We see the orange being peeled, but we do not see the underlying control signals that caused the movement. LfV methods must therefore find ways to infer these "missing actions" or learn without them.
A Methodological Framework for LfV
The survey categorizes current research into three primary pillars that define the LfV pipeline:
1. Knowledge Extraction
This pillar focuses on pulling foundational information from video without requiring direct robot interaction. Researchers use large-scale video to pre-train visual representations that understand spatial relationships, object permanence, and physical properties. These representations serve as a "common sense" baseline for the robot.
2. Overcoming LfV Challenges
To bridge the human-to-robot gap, researchers employ two main strategies:
- Domain Adaptation: Techniques that make robotic data look more like human video (or vice versa) to minimize the visual distribution shift.
- Action Inference: The development of models that attempt to "guess" the actions taken in a video. By generating pseudo-labels for passive video, researchers can transform internet data into a format that resembles traditional robotic demonstrations.
3. Video-Informed Training
This is the final stage where extracted knowledge is integrated into robot control. Video data is used to improve policy learning (the decision-making process) and dynamics models (the robot's ability to predict how the environment will change in response to its actions). By watching video, robots can learn to predict the future state of the world, effectively using the internet as a massive physics simulator.
Key Insights: Video as a Physics Engine
A central insight of the survey is that internet video is more than a collection of images; it is a rich repository of physical dynamics. By observing millions of interactions, models can learn the "intuitive physics" of our world—such as the fact that objects fall downward or that a door handle must be turned before the door can be pulled.
The authors argue that the transition from "passive observation" to "active embodiment" is the most critical research frontier. Solving the action gap is the key to unlocking the power of the internet for robotics, allowing models to move from simply recognizing tasks to actually performing them.
Future Directions and the Path to Generalization
The survey concludes by highlighting that while we have made strides in representation learning, we have yet to fully scale these methods into truly generalist robot policies. The authors identify three critical areas for future work:
- Scalable Foundation Models: There is a pressing need for architectures that can handle the sheer diversity of internet video without requiring task-specific fine-tuning.
- Enhanced Dynamics Modeling: Using video to help robots predict the future is a burgeoning area that could prove more effective than simple imitation learning.
- General-Purpose Robotics: The ultimate goal is a robot that can perform tasks it was never explicitly programmed for by drawing on its vast "knowledge" of world dynamics extracted from video.
By synthesizing the current state of the art, this survey provides a roadmap for using the world's digital video library to build the next generation of intelligent, capable, and general-purpose robots.
Summary updated