Scaling Robotic Learning Beyond Expert Demonstrations

2,149 concepts1,616 papers39 proposals

The "Transformer revolution" has dramatically reshaped natural language processing and computer vision by training massive foundation models on internet-scale text and image datasets. However, as noted by Zhiyuan Xu et al. in A Survey on Robotics with Foundation Models: toward Embodied AI, a massive disparity exists between these abundant web-scale corpora and the scarcity of high-quality robotic interaction trajectories.

Why is robotic data so hard to scale? In language or vision, data is passive and readily available. In contrast, training general-purpose robots requires what Huy Ha et al. term "robot-complete" data in Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition—synchronized streams of video observations, precise motor actions, and semantic language labels. Traditionally, this data is gathered via human teleoperation, where an expert physically guides a robot through a task. As highlighted by Murtaza Dalal et al. in Imitating Task and Motion Planning with Visuomotor Transformers, this process is slow, expensive, and fundamentally non-scalable. This expert demonstration bottleneck restricts models to narrow environments and specific hardware, preventing the emergence of broad generalization seen in other domains (Songming Liu et al. in RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation).

To build true foundation models for physical control, researchers are developing techniques to bypass, augment, or reinvent the data collection pipeline.

Bridging the Embodiment Gap: Learning from Action-Free Human Videos

If collecting robot-specific data is the bottleneck, the most intuitive solution is to leverage the vast amount of human activity already recorded in video format. However, this introduces the "embodiment gap": human hands do not move like robotic grippers, and passive videos lack the precise joint-torque or velocity labels (actions) required to train a motor policy.

To bridge this gap, researchers decouple semantic understanding from low-level execution. For instance, in PLEX: Making the Most of the Available Data for Robotic Manipulation Pretraining, Garrett Thomas et al. separate high-level task planning from low-level motor execution, allowing the model to learn what to do from video-only data and how to do it from task-agnostic robot trajectories.

Another approach is to extract abstract representations that ignore the specific embodiment. In Structured World Models from Human Videos, Shikhar Bahl et al. introduce a "morphology-invariant" action space based on visual affordances (e.g., identifying where an object should be grasped), enabling robots to learn from human videos with minimal real-world interaction data. Similarly, Jaden Clark et al. develop RAD (Reasoning through Action-free Data) in Action-Free Reasoning for Policy Generalization, which extracts language-based "reasoning chains" from human videos, allowing the policy to co-train on action-rich robot data and reasoning-rich human data.

Alternatively, generative AI can be used to translate human videos directly into the robot's domain. In MimicDreamer: Aligning Human and Robot Demonstrations for Scalable VLA Training, Haoyun Li et al. use video diffusion models to visually swap human hands with robotic arms, transforming cheap internet videos into high-quality training data. Rather than predicting joint angles, the authors of Generalist Robot Manipulation beyond Action Labeled Data predict the 3D motion of hands and grippers as dynamic point clouds, allowing the system to learn from unlabeled videos of both humans and robots.

Scaling via High-Fidelity Simulation and Generative AI

While human videos provide semantic breadth, robots still need to learn the physics of contact dynamics. To scale this data safely and rapidly, researchers are turning to automated simulation pipelines.

The power of simulation lies in its ability to generate thousands of diverse environments and trajectories automatically. In RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots, Soroush Nasiriany et al. leverage generative AI to populate simulated kitchens with thousands of 3D assets and over 100,000 trajectories, demonstrating that massive synthetic datasets yield strong scaling trends in real-world capabilities. To ensure these policies transfer successfully to the real world (Sim2Real), Guiliang Liu et al. introduce DexScale in DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Control, which projects human demonstrations into simulation and uses large language models to automatically scale domain randomization.

Simulation also allows for the generation of massive, highly specialized datasets that would be impossible to collect manually. For example, in DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation, the authors utilize a differentiable optimization pipeline to generate 1.32 million valid grasps for multi-fingered hands, providing the scale necessary to train robust deep learning models for high-degree-of-freedom manipulation.

Democratizing and Reannotating Real-World Collection

When real-world data collection is unavoidable, the goal shifts to lowering the hardware barrier and maximizing the utility of every collected trajectory.

Traditional teleoperation requires expensive, specialized rigs. To democratize this process, platforms like ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation enable users to teleoperate robots remotely using standard smartphones as 6-DoF controllers, drastically increasing data collection throughput. Similarly, Hongjie Fang et al. introduce AirExo in Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wild, a low-cost, 3D-printed exoskeleton that allows users to collect "in-the-wild" human demonstrations that can be mixed with limited robot data. For complex mobile manipulation, frameworks like TeleMoMa (TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation) unify diverse, accessible interfaces like RGB-D cameras and VR controllers to simplify whole-body data collection.

Finally, if the collected data is noisy or completely action-free, algorithmic reannotation can salvage its utility. In Learning to Drive Anywhere with Model-Based Reannotation, Noriaki Hirose et al. propose Model-Based ReAnnotation (MBRA). MBRA uses a model-based expert to compute precise training labels for low-quality, crowd-sourced, or passive data, turning otherwise unusable videos into robust navigation policies.

By combining action-free video pre-training, automated simulation scaling, and low-cost real-world collection, modern robotics is steadily breaking through its data bottleneck, paving the way for true generalist physical agents.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_robotic_data_scarcity_f3199407,
  title = {robotic data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robotic-data-scarcity/rdr_f31994079df545bf8189e8e8135a2dda},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_robotic_data_scarcity_f3199407_v1,
  title = {robotic data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robotic-data-scarcity/rdr_f31994079df545bf8189e8e8135a2dda/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake