Towards Generalist Robot Learning from Internet Video: A Survey (Abstract Reprint)

Robert McCarthy, Daniel Chee Hian Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas George Thuruthel, Zhibin Li

TL;DR

Robot learning is currently bottlenecked by a lack of diverse, large-scale data. This survey explores the emerging field of Learning from Videos (LfV), which aims to bridge this gap by leveraging the vast amount of human and physical interaction data available on the internet to train general-purpose robots.

What Problem Does This Paper Address?

While Large Language Models (LLMs) and generative video models have scaled exponentially by "eating the internet," robotics has lagged behind. Training a robot typically requires high-quality, robot-specific data—often collected through expensive teleoperation or slow simulations. This creates a massive data bottleneck.

The authors address the challenge of how we can use the billions of hours of existing internet video (YouTube, TikTok, etc.) to teach robots about the world. However, using this data isn't straightforward. Internet videos suffer from two primary issues:

  1. Distribution Shift: A human hand peeling an orange in a kitchen looks and moves very differently than a metal robotic gripper in a lab.
  2. Missing Action Labels: Unlike robot demonstration data, internet videos don't come with the underlying "motor commands" (joint torques or velocities) that produced the movement. We see the effect, but not the cause.

Methods

As a survey paper, the authors provide a systematic taxonomy of how researchers are currently tackling LfV. They categorize the field into three main pillars:

  • Knowledge Extraction: Techniques for pulling "foundational" information from video, such as pre-training visual representations that understand physical properties and spatial relationships.
  • Overcoming LfV Challenges: Methods designed to handle the "human-to-robot" gap. This includes domain adaptation (making robot data look like human data or vice versa) and Action Inference, where models attempt to "guess" the actions taken in a video to create pseudo-labels for training.
  • Video-Informed Training: The actual integration of this knowledge into robot control. This involves using video data to improve policy learning (how the robot decides what to do) and dynamics models (how the robot predicts the world will change).

Datasets

While this specific abstract reprint does not list every individual dataset covered in the full 48-page journal article, it highlights the shift toward large-scale internet video as the primary data source.

The paper contrasts these massive, "in-the-wild" video sources against "traditional robot data," which is typically smaller, task-specific, and collected in controlled environments. The goal of LfV is to use the former to augment the latter.

Key Insights and Ideas

  • Video as a Physics Engine: Internet video is more than just pictures; it is a rich repository of "foundational information regarding physical dynamics." By watching enough video, a model can learn the "common sense" of physics—e.g., that objects fall down, or that a door must be pulled by its handle.
  • The Action Gap is the Main Hurdle: The transition from "passive observation" (watching a video) to "active embodiment" (moving a robot arm) is the core technical challenge. Solving the "missing action label" problem is seen as the key to unlocking the power of the internet for robotics.
  • Foundation Models for Robotics: The authors argue that the future of the field lies in scalable foundation models that can process the full spectrum of internet video to enhance both robot policies and world models.

Key Results

The paper serves as a comprehensive roadmap rather than a single experimental report. Its "results" are the synthesis of the current state of the art:

  • Taxonomy of LfV: It provides a structured framework for understanding how different research threads (representation learning, domain adaptation, and imitation learning) fit together.
  • Identification of Research Gaps: The survey highlights that while we are getting better at extracting features from video, we still struggle to scale these methods into truly generalist robot policies that work across different hardware and environments.

Limitations and Future Work

The authors identify several critical areas for future research:

  • Scalability: Current methods often work in niche settings. There is a pressing need for "scalable foundation model approaches" that can handle the sheer diversity of the internet without breaking.
  • Dynamics Modeling: Using video to help robots predict the future (dynamics) is a burgeoning area that needs more focus compared to simple imitation.
  • Generalization: The ultimate goal remains the "general-purpose robot." The authors emphasize that LfV is the most promising path toward robots that can perform tasks they were never explicitly programmed for, simply by drawing on their "knowledge" of how the world works.

Summary updated

Abstract

Scaling deep learning to massive and diverse internet data has driven remarkable breakthroughs in domains such as video generation and natural language processing. Robot learning, however, has thus far failed to replicate this success and remains constrained by a scarcity of available data. Learning from Videos (LfV) methods aim to address this data bottleneck by augmenting traditional robot data with large-scale internet video. This video data provides foundational information regarding physical dynamics, behaviours, and tasks, and can be highly informative for general-purpose robots. This survey systematically examines the emerging field of LfV. We first outline essential concepts, including detailing fundamental LfV challenges such as distribution shift and missing action labels in video data. Next, we comprehensively review current methods for extracting knowledge from large-scale internet video, overcoming LfV challenges, and improving robot learning through video-informed training. The survey concludes with a critical discussion of future opportunities. Here, we emphasize the need for scalable foundation model approaches that can leverage the full range of available internet video and enhance the learning of robot policies and dynamics models. Overall, the survey aims to inform and catalyse future LfV research, driving progress towards general-purpose robots.
BibTeX
@inproceedings{mccarthy2026towards,
  author = {Robert McCarthy and Daniel Chee Hian Tan and Dominik Schmidt and Fernando Acero and Nathan Herr and Yilun Du and Thomas George Thuruthel and Zhibin Li},
  title = {{Towards Generalist Robot Learning from Internet Video: A Survey (Abstract Reprint)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i47.41397},
}

Loading related content...

Figure