Dual System Architectures for Real Time Robotic Control

This direction is split into other directions: inference latency bottleneck (VLM inference latency). Its last version stays readable and citable here.

Version 1 of 1448 papers492 conceptsfrozen 2026-10-04

Vision-Language-Action (VLA) models have revolutionized robotics by allowing agents to understand open-ended instructions like "pick up the low-calorie drink." However, these models face a fundamental "thinking-acting" gap. While a large model might take hundreds of milliseconds to "think" (inference), a robot needs to "act" (control loop) at frequencies of 50Hz to 100Hz to remain stable and reactive.

The emerging consensus across recent research is that we cannot simply wait for larger models to get faster. Instead, the field is moving toward dual-system architectures that decouple high-level semantic reasoning from low-level motor execution.

The Latency Bottleneck of Autoregressive Reasoning

Traditional VLAs treat robotic control as a translation task: pixels and text go in, and discrete action tokens come out. As Moo Hyun Kim and colleagues point out in Optimized Fine-Tuning (OFT), this autoregressive decoding is inherently slow. Generating actions token-by-token limits the control frequency to just a few hertz, which is insufficient for bimanual manipulation or dynamic environments where a robot must react to a slipping object in milliseconds.

This creates a trade-off: use a massive model for "common sense" but suffer jerky, slow movements, or use a small, fast model that is reactive but "dumb" and fails at novel instructions.

Decoupling System 1 and System 2

To solve this, researchers are drawing inspiration from cognitive science, specifically the "System 1" (fast, intuitive) and "System 2" (slow, deliberative) framework.

In FiS-VLA, Hao Chen and others propose embedding a high-speed execution module directly into the final layers of a large VLM. This allows the model to share semantic features while running the execution head at a blazing 117.7 Hz. Similarly, ByungOk Han introduces DP-VLA, which explicitly splits control into a "slow" reasoning process and a "fast" motor process, achieving an 8x speedup over monolithic models.

The intuition is simple: the robot doesn't need to re-evaluate the entire history of the task 100 times a second. It only needs to "re-think" the plan occasionally, while the "fast" system handles the immediate physical adjustments.

From Discrete Tokens to Continuous Flow

Another major shift involves changing how the model outputs actions. Instead of predicting the next discrete token, newer models like $\pi_0$ by Kevin Black and Noah Brown use flow matching.

Flow matching allows the model to generate continuous action trajectories in a single step rather than iteratively. This is often combined with Action Chunking, where the model predicts a sequence of future actions (a "chunk") all at once. To prevent the robot from pausing between chunks, Kevin Black also introduced Real-Time Chunking (RTC), which uses diffusion inpainting to smoothly blend the current action with the next predicted chunk, ensuring fluid motion even when the "brain" is lagging.

Knowledge Insulation and Efficiency

A final challenge is ensuring that making a model "fast" doesn't make it "forget." When fine-tuning a VLM to become a VLA, the new action-related gradients can "corrupt" the pre-trained semantic knowledge.

Danny Driess and brian ichter address this in Knowledge Insulating VLAs. They use a stop-gradient mechanism to "insulate" the VLM's core knowledge while training a specialized continuous action expert. This allows the model to run at high frequencies and generalize to new tasks without losing the "common sense" it gained from internet-scale pre-training.

By combining these strategies—hierarchical dual-systems, continuous action heads, and knowledge insulation—the field is moving toward agents that can both reason about complex human worlds and move with the grace and speed required to inhabit them.

Papers in this version

Showing 60 of 448 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dual_system_architectures_for_real_time__c82102b2,
  title = {Dual System Architectures for Real Time Robotic Control},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dual-system-architectures-for-real-time-robotic-control/rdr_c82102b2151749abab60bce5d16b173a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dual_system_architectures_for_real_time__c82102b2_v1,
  title = {Dual System Architectures for Real Time Robotic Control},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dual-system-architectures-for-real-time-robotic-control/rdr_c82102b2151749abab60bce5d16b173a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}