Scaling General Purpose Robotics via Vision Language Action Models

This direction is merged into another direction: robotic data scarcity. Its last version stays readable and citable here.

Version 1 of 11,151 papers1,298 conceptsfrozen 2026-10-04

The "Transformer revolution" that redefined natural language and computer vision has arrived in robotics, but it faces a fundamental obstacle: the data bottleneck. While large language models (LLMs) train on trillions of tokens scraped from the internet, robots require high-quality, paired perception-action trajectories. Collecting these through manual teleoperation is slow, expensive, and difficult to scale.

To solve this, researchers are developing Vision-Language-Action (VLA) models. These are foundation models for embodied AI that map visual observations and natural language instructions directly into physical motor commands. By reframing robotic control as a multimodal sequence-to-sequence problem, VLAs aim to inherit the reasoning and generalization capabilities of internet-scale models.

The Data Bottleneck in Embodied AI

The primary challenge identified across the field—from Anthony Brohan to Chelsea Finn—is that robotic data is "expensive." Unlike text, which is a byproduct of human digital life, robot trajectories must be intentionally created.

RT-1: Robotics Transformer for Real-World Control at Scale demonstrated that a Transformer-based architecture could achieve high performance when trained on 130,000 real-world demonstrations. However, as Soroush Nasiriany and others note, even 100,000 trajectories are a drop in the bucket compared to the datasets used for GPT-4. To achieve true general-purpose robotics, the field must find ways to scale data without relying solely on human teleoperation.

From Semantics to Actuation

The breakthrough in RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control was the realization that robot actions can be represented as text tokens. By co-fine-tuning a massive Vision-Language Model (VLM) on robotic data, Brianna Zitkovich and her team showed that the model could "transfer" its semantic understanding of the world (e.g., knowing what a "dinosaur" is from the internet) to physical tasks (e.g., picking up a toy dinosaur it has never seen in a robot lab).

Other approaches like LLaRA by Xiang Li further this by aligning actions with image pixel coordinates, effectively treating the robot's "thoughts" about movement as a conversation between the model and the environment.

Bridging the Embodiment Gap

If robot-specific data is scarce, can we use the billions of hours of human video available on the internet? The difficulty lies in the "embodiment gap": a human hand looks and moves differently than a robotic gripper.

Several strategies have emerged to bridge this:

  • Masking and Invariants: Jiajun Wu and Chelsea Finn found that simply masking the human hand in videos allows robots to focus on the object's movement, leading to a 58% improvement in success rates.
  • Latent Action Spaces: Instead of predicting raw motor torques, models like LAPA by Seonghyeon Ye and IGOR learn a discrete "latent action" space. This space acts as a universal language of motion shared by both humans and robots.
  • Action-Free Reasoning: Jaden Clark and Dorsa Sadigh proposed RAD, which extracts language-based "reasoning chains" from human videos. By learning why a human performs an action, the robot can better generalize to unseen tasks.

Scaling through Synthesis and Simulation

When real-world data is insufficient, researchers turn to generative models to synthesize experience. GigaBrain-0 uses a generative world model to create photorealistic "Real2Real" transfers, while Real2Render2Real by Justin Yu transforms a single smartphone video of a human into thousands of synthetic robot executions.

Simulation frameworks like RoboCasa and InfiniteWorld provide the infrastructure for "scaling laws" in robotics. These environments allow agents to practice in diverse, AI-enriched scenes, proving that synthetic trajectories can significantly boost real-world performance.

Collaborative Intelligence

As these models grow, the architecture itself is evolving to handle complexity. AdaMoE, introduced by Weijie Shen, uses a Mixture of Experts (MoE) approach. Unlike standard MoE where one expert "wins," AdaMoE allows multiple experts to collaborate on a single action. This decoupling ensures that specialized knowledge—such as grasping versus navigating—can be combined fluidly, allowing the model to handle the multi-faceted nature of physical interaction.

By combining internet-scale semantic knowledge, latent motion representations, and massive synthetic data generation, VLA models are transforming robots from task-specific tools into general-purpose agents capable of understanding and acting upon the world.

Papers in this version

Showing 60 of 1,151 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scaling_general_purpose_robotics_via_vis_3311a047,
  title = {Scaling General Purpose Robotics via Vision Language Action Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-general-purpose-robotics-via-vision-language-action-models/rdr_3311a04747d548d080741f58c17c8ffa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_general_purpose_robotics_via_vis_3311a047_v1,
  title = {Scaling General Purpose Robotics via Vision Language Action Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-general-purpose-robotics-via-vision-language-action-models/rdr_3311a04747d548d080741f58c17c8ffa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}