TL;DR
Robot learning is currently bottlenecked by a lack of diverse, large-scale data. This survey explores the emerging field of Learning from Videos (LfV), which aims to bridge this gap by leveraging the vast amount of human and physical interaction data available on the internet to train general-purpose robots.
What Problem Does This Paper Address?
While Large Language Models (LLMs) and generative video models have scaled exponentially by "eating the internet," robotics has lagged behind. Training a robot typically requires high-quality, robot-specific data—often collected through expensive teleoperation or slow simulations. This creates a massive data bottleneck.
The authors address the challenge of how we can use the billions of hours of existing internet video (YouTube, TikTok, etc.) to teach robots about the world. However, using this data isn't straightforward. Internet videos suffer from two primary issues:
- Distribution Shift: A human hand peeling an orange in a kitchen looks and moves very differently than a metal robotic gripper in a lab.
- Missing Action Labels: Unlike robot demonstration data, internet videos don't come with the underlying "motor commands" (joint torques or velocities) that produced the movement. We see the effect, but not the cause.
Methods
As a survey paper, the authors provide a systematic taxonomy of how researchers are currently tackling LfV. They categorize the field into three main pillars:
- Knowledge Extraction: Techniques for pulling "foundational" information from video, such as pre-training visual representations that understand physical properties and spatial relationships.
- Overcoming LfV Challenges: Methods designed to handle the "human-to-robot" gap. This includes domain adaptation (making robot data look like human data or vice versa) and Action Inference, where models attempt to "guess" the actions taken in a video to create pseudo-labels for training.
- Video-Informed Training: The actual integration of this knowledge into robot control. This involves using video data to improve policy learning (how the robot decides what to do) and dynamics models (how the robot predicts the world will change).
Datasets
While this specific abstract reprint does not list every individual dataset covered in the full 48-page journal article, it highlights the shift toward large-scale internet video as the primary data source.
The paper contrasts these massive, "in-the-wild" video sources against "traditional robot data," which is typically smaller, task-specific, and collected in controlled environments. The goal of LfV is to use the former to augment the latter.
Key Insights and Ideas
- Video as a Physics Engine: Internet video is more than just pictures; it is a rich repository of "foundational information regarding physical dynamics." By watching enough video, a model can learn the "common sense" of physics—e.g., that objects fall down, or that a door must be pulled by its handle.
- The Action Gap is the Main Hurdle: The transition from "passive observation" (watching a video) to "active embodiment" (moving a robot arm) is the core technical challenge. Solving the "missing action label" problem is seen as the key to unlocking the power of the internet for robotics.
- Foundation Models for Robotics: The authors argue that the future of the field lies in scalable foundation models that can process the full spectrum of internet video to enhance both robot policies and world models.
Key Results
The paper serves as a comprehensive roadmap rather than a single experimental report. Its "results" are the synthesis of the current state of the art:
- Taxonomy of LfV: It provides a structured framework for understanding how different research threads (representation learning, domain adaptation, and imitation learning) fit together.
- Identification of Research Gaps: The survey highlights that while we are getting better at extracting features from video, we still struggle to scale these methods into truly generalist robot policies that work across different hardware and environments.
Limitations and Future Work
The authors identify several critical areas for future research:
- Scalability: Current methods often work in niche settings. There is a pressing need for "scalable foundation model approaches" that can handle the sheer diversity of the internet without breaking.
- Dynamics Modeling: Using video to help robots predict the future (dynamics) is a burgeoning area that needs more focus compared to simple imitation.
- Generalization: The ultimate goal remains the "general-purpose robot." The authors emphasize that LfV is the most promising path toward robots that can perform tasks they were never explicitly programmed for, simply by drawing on their "knowledge" of how the world works.
Summary updated