Towards Generalist Robot Learning from Internet Video: A Survey (Abstract Reprint)

Robert McCarthy, Daniel C.h. Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas G. Thuruthel, Zhibin Li

Introduction

In recent years, deep learning has achieved transformative breakthroughs in fields like natural language processing and video generation by scaling models on massive, diverse internet datasets. However, robotics has yet to experience a similar "scaling moment." The primary constraint remains a severe data bottleneck: while text and images are abundant, high-quality robotic data—typically gathered through labor-intensive teleoperation or slow, specialized simulations—is scarce and expensive to produce.

Learning from Videos (LfV) has emerged as a promising paradigm to bridge this gap. By leveraging the billions of hours of human activity and physical interactions available on platforms like YouTube and TikTok, researchers aim to provide robots with a foundational understanding of the world. This survey systematically examines how internet video can be used to teach robots about physical dynamics, behaviors, and complex tasks, moving the field closer to the goal of general-purpose embodied intelligence.

The Core Challenges of LfV

While internet video is a goldmine of information, it cannot be directly ingested by robotic controllers due to two fundamental discrepancies:

  1. Distribution Shift (The Embodiment Gap): There is a significant visual and physical mismatch between video data and robotic execution. A human hand peeling an orange in a dimly lit kitchen looks and moves differently than a metallic robotic gripper in a controlled laboratory environment. This shift makes it difficult for a robot to map what it sees to its own physical form.
  2. Missing Action Labels: This is perhaps the most significant technical hurdle. Unlike robot demonstration data, which records specific motor commands (joint torques or velocities), internet videos only show the results of actions. We see the orange being peeled, but we do not see the underlying control signals that caused the movement. LfV methods must therefore find ways to infer these "missing actions" or learn without them.

A Methodological Framework for LfV

The survey categorizes current research into three primary pillars that define the LfV pipeline:

1. Knowledge Extraction

This pillar focuses on pulling foundational information from video without requiring direct robot interaction. Researchers use large-scale video to pre-train visual representations that understand spatial relationships, object permanence, and physical properties. These representations serve as a "common sense" baseline for the robot.

2. Overcoming LfV Challenges

To bridge the human-to-robot gap, researchers employ two main strategies:

  • Domain Adaptation: Techniques that make robotic data look more like human video (or vice versa) to minimize the visual distribution shift.
  • Action Inference: The development of models that attempt to "guess" the actions taken in a video. By generating pseudo-labels for passive video, researchers can transform internet data into a format that resembles traditional robotic demonstrations.

3. Video-Informed Training

This is the final stage where extracted knowledge is integrated into robot control. Video data is used to improve policy learning (the decision-making process) and dynamics models (the robot's ability to predict how the environment will change in response to its actions). By watching video, robots can learn to predict the future state of the world, effectively using the internet as a massive physics simulator.

Key Insights: Video as a Physics Engine

A central insight of the survey is that internet video is more than a collection of images; it is a rich repository of physical dynamics. By observing millions of interactions, models can learn the "intuitive physics" of our world—such as the fact that objects fall downward or that a door handle must be turned before the door can be pulled.

The authors argue that the transition from "passive observation" to "active embodiment" is the most critical research frontier. Solving the action gap is the key to unlocking the power of the internet for robotics, allowing models to move from simply recognizing tasks to actually performing them.

Future Directions and the Path to Generalization

The survey concludes by highlighting that while we have made strides in representation learning, we have yet to fully scale these methods into truly generalist robot policies. The authors identify three critical areas for future work:

  • Scalable Foundation Models: There is a pressing need for architectures that can handle the sheer diversity of internet video without requiring task-specific fine-tuning.
  • Enhanced Dynamics Modeling: Using video to help robots predict the future is a burgeoning area that could prove more effective than simple imitation learning.
  • General-Purpose Robotics: The ultimate goal is a robot that can perform tasks it was never explicitly programmed for by drawing on its vast "knowledge" of world dynamics extracted from video.

By synthesizing the current state of the art, this survey provides a roadmap for using the world's digital video library to build the next generation of intelligent, capable, and general-purpose robots.

Summary updated

Abstract

Scaling deep learning to massive and diverse internet data has driven remarkable breakthroughs in domains such as video generation and natural language processing. Robot learning, however, has thus far failed to replicate this success and remains constrained by a scarcity of available data. Learning from Videos (LfV) methods aim to address this data bottleneck by augmenting traditional robot data with large-scale internet video. This video data provides foundational information regarding physical dynamics, behaviours, and tasks, and can be highly informative for general-purpose robots. This survey systematically examines the emerging field of LfV. We first outline essential concepts, including detailing fundamental LfV challenges such as distribution shift and missing action labels in video data. Next, we comprehensively review current methods for extracting knowledge from large-scale internet video, overcoming LfV challenges, and improving robot learning through video-informed training. The survey concludes with a critical discussion of future opportunities. Here, we emphasize the need for scalable foundation model approaches that can leverage the full range of available internet video and enhance the learning of robot policies and dynamics models. Overall, the survey aims to inform and catalyse future LfV research, driving progress towards general-purpose robots.
BibTeX
@inproceedings{mccarthy2026towards,
  author = {Robert McCarthy and Daniel C.H. Tan and Dominik Schmidt and Fernando Acero and Nathan Herr and Yilun Du and Thomas G. Thuruthel and Zhibin Li},
  title = {{Towards Generalist Robot Learning from Internet Video: A Survey (Abstract Reprint)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i47.41397},
}
Figure