Learning from Watching: Scalable Extraction of Manipulation Trajectories from Human Videos

Xiaowei Hu, Guang YearXiv 2025 · DOI · arXiv

TL;DR

Researchers have developed a two-stage framework that transforms ordinary human manipulation videos into high-quality training data for robots. By combining the semantic "reasoning" of large foundation models (like GPT-4o) with the precision of dense point-tracking algorithms, the system can extract consistent trajectories for hands, tools, and objects even through complex occlusions.

What Problem Does This Paper Address?

Training a robot to perform complex tasks—like sautéing vegetables or using a screwdriver—requires massive amounts of data. Traditionally, this data comes from teleoperation (a human remotely controlling a robot) or scripted demonstrations. Both are notoriously slow, expensive, and difficult to scale.

The internet is full of "in-the-wild" human demonstration videos (YouTube, TikTok, etc.) that could theoretically solve this data bottleneck. However, turning raw video into robot-readable trajectories is hard. Current methods often focus only on the hand or only on the object, and they frequently "lose" the target when it's hidden behind something else (occlusion). This paper tackles the challenge of extracting dense, temporally consistent, and semantically meaningful trajectories from unconstrained video.

Methods

The authors propose a two-stage pipeline that bridges high-level "understanding" with low-level "tracking":

  1. Semantic Keypoint Proposal: The system uses a Large Foundation Model (LFM), specifically GPT-4o, to analyze a clear frame from a video. Guided by category-specific prompts, the LFM identifies "semantic landmarks"—such as fingertips, tool tips, and object corners. This provides the "what" and "where" for the start of the task.
  2. Dense Trajectory Recovery: These keypoints are fed into LocoTrack, a dense point-tracking network. Unlike simple box-trackers, this tracks specific pixels across the entire video sequence.
  3. Refinement & Stability: To ensure the data is clean enough for robot training, the framework employs:
  4. Bidirectional Tracking: Tracking both forward and backward in time to check for consistency.
  5. Interpolation: Filling in the gaps when a keypoint is briefly hidden from view.
  6. Standardization: Mapping human wrist trajectories to a generic robot end-effector model, making the data "embodiment-agnostic" (usable by different types of robots).

Datasets

The study evaluated the performance of several LFMs (GPT-4o, Gemini 1.5, Grok V, and Claude 3 Opus) using publicly available datasets containing annotated keypoints to determine which model was best at identifying manipulation landmarks. For the tracking phase, they validated the framework on a diverse collection of real-world human manipulation videos featuring various tasks and camera angles.

Key Insights and Ideas

  • The "Brain-Eye" Synergy: The core insight is that foundation models are excellent at identifying what is important (semantics), while point trackers are excellent at following pixels (dynamics). Combining them overcomes the limitations of using either in isolation.
  • Occlusion Robustness: By using dense point tracking and bidirectional consistency, the system can "remember" where a tool or finger is even when it's temporarily covered by a hand or an object—a major failure point for previous methods like ViViDex.
  • Scalability via Abstraction: By mapping human motion to a standardized end-effector trajectory, the authors create a pipeline that can turn almost any YouTube tutorial into a "robot masterclass" without needing specialized hardware or manual labeling.

Key Results

The researchers compared their framework against three major baselines: ViViDex, VidBot, and DexMV.

  • Versatility: Their method was the only one to support all four critical criteria: multi-object tracking, occlusion robustness, trajectory prediction, and compatibility with arbitrary (unconstrained) videos.
  • LFM Performance: GPT-4o was identified as the superior foundation model for this task, showing "outstanding accuracy" in recognizing semantic keypoints compared to its peers.
  • Qualitative Success: The system successfully recovered complex motion trajectories with high temporal consistency across various real-world manipulation tasks.

Limitations and Future Work

While the paper presents a significant step forward, some limitations remain:

  • Initial Detection Dependency: The system relies on the LFM correctly identifying keypoints in the initial frames. If the LFM misses a crucial tool tip at the start, the tracker has nothing to follow.
  • 2D to 3D Gap: While the trajectories are dense and consistent in 2D, translating these perfectly into 3D space for robot execution remains a complex challenge.

Future Work: The authors envision using this framework to build massive, internet-scale datasets for pretraining robot policies, potentially leading to robots that can learn new skills simply by "watching" a few videos.

Summary updated

Abstract

Collecting high-quality data for training large-scale robotic models typically relies on real robot platforms, which is labor-intensive and costly, whether via teleoperation or scripted demonstrations. To scale data collection, many researchers have turned to leveraging human manipulation videos available online. However, current methods predominantly focus on hand detection or object pose estimation, failing to fully exploit the rich interaction cues embedded in these videos. In this work, we propose a novel approach that combines large foundation models for video understanding with point tracking techniques to extract dense trajectories of all task-relevant keypoints during manipulation. This enables more comprehensive utilization of Internet-scale human demonstration videos. Experimental results demonstrate that our method can accurately track keypoints throughout the entire manipulation process, paving the way for more scalable and data-efficient robot learning.
BibTeX
@misc{hu2025learning,
  author = {Xiaowei Hu and Guang Ye},
  title = {{Learning from Watching: Scalable Extraction of Manipulation Trajectories from Human Videos}},
  year = {2025},
  eprint = {2512.00024},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.00024},
  howpublished = {arXiv},
}
Figure