When we deploy large foundation models in digital environments—such as generating text or writing code—a delay of a few hundred milliseconds is barely noticeable. However, when a model is deployed as the "brain" of a physical robot, that same delay can be catastrophic.
Physical systems operate under strict real-time control constraints. To maintain balance, react to sudden environmental changes, or perform fluid manipulation, a robot's internal control loop must typically run at frequencies between 100 Hz and 1000 Hz. This leaves a strict time budget of only 1 to 10 milliseconds per control step. As Zhiyuan Xu et al. note, a fundamental conflict exists between the massive computational requirements of foundation models and the low-latency execution required for real-time safety and stability. If a model's inference time exceeds this millisecond-level window, execution gaps emerge, leading to jerky movements, instability, or system crashes.
The Root of the Latency Problem
To understand why modern vision-language-action (VLA) models and generative policies struggle to meet these constraints, we must look at how they compute actions. The latency bottleneck stems primarily from three architectural paradigms:
- Autoregressive Token Generation: Many state-of-the-art VLAs treat actions as discrete text tokens. To output a 7-degree-of-freedom joint command, the model must predict each dimension sequentially, one token at a time. As Danny Driess et al. and Jianwei Yang et al. highlight, this autoregressive decoding requires multiple sequential forward passes through billions of parameters for a single control step, making real-time execution mathematically prohibitive.
- Iterative Diffusion Processes: Diffusion policies excel at modeling complex, multimodal action distributions. However, they generate actions by starting with random noise and iteratively refining it over dozens of denoising steps. Yanjiang Guo et al. and Michael Drolet et al. observe that this iterative sampling creates a massive computational burden that severely limits the operational frequency of the robot.
- Quadratic History Scaling: Robots rely on historical context (past states and images) to handle partially observable environments. In standard Transformer backbones, the computational cost of processing this history scales quadratically with the sequence length. Yueh-Hua Wu shows that evaluating varying history lengths during action selection quickly violates strict real-time timing constraints.
This creates a stark trade-off. As Scott E. Reed argues, increasing a generalist agent's parameter count improves its reasoning and generalization, but directly harms its ability to operate in real-time physical environments.
Amortizing Latency via Action Chunking
Instead of forcing a massive model to think faster, what if it didn't have to think as often? This is the core intuition behind action chunking.
Standard Policy: [Observe] -> [Think (Slow)] -> [Execute 1 Step] -> [Observe] -> [Think (Slow)]
Action Chunking: [Observe] -> [Think (Slow)] -> [Execute Step 1] -> [Execute Step 2] -> [Execute Step 3]
└─────────── Amortized Latency Window ───────────┘
Instead of predicting a single action for the immediate next timestep, the model predicts a "chunk" or sequence of future actions (e.g., the next 10 to 20 steps) simultaneously. The robot's low-level hardware controller can then execute this sequence step-by-step without waiting for the large model. This effectively amortizes the heavy inference cost over a longer temporal window.
Xinyu Zhang et al. leverage this via "chunking autoregression," predicting variable-sized blocks of actions in a single step to maintain temporal smoothness while bypassing token-by-token delays. To make this process even more efficient, Hongyi Zhou et al. introduce BEAST, which encodes continuous trajectories into compact, fixed-length tokens using B-splines. Because B-splines mathematically guarantee smooth transitions between action sequences, the model can utilize parallel decoding to bypass autoregressive bottlenecks entirely, satisfying high-frequency hardware requirements.
Distillation and Constant-Time Architectures
Another prominent direction focuses on compressing the model's temporal footprint, either by compressing the generation process or changing the underlying network architecture.
Consistency Distillation
To bring the expressive power of diffusion policies to real-time systems, researchers use consistency distillation to collapse multi-step iterative sampling into a single step. Aaditya Prasad proposes Consistency Policy, a framework that distills a pre-trained diffusion policy into a model that can predict high-quality action distributions in just one to three steps. This achieves a 10x to 100x speedup, making generative visuomotor control feasible on resource-constrained edge hardware.
Linear-Time Recurrent Models
To solve the quadratic slowdown caused by long historical contexts in Transformers, researchers are exploring alternative sequence-modeling backbones. Thomas Schmied et al. introduce the Large Recurrent Action Model (LRAM), which replaces the Transformer backbone with an xLSTM architecture.
Unlike Transformers, which must look back across an explicit history buffer at every step, xLSTM maintains a fixed-size internal state that tracks historical context implicitly. This allows for linear-time inference ($O(1)$ per timestep), ensuring that the model's execution speed remains constant whether the robot has been running for five seconds or five hours.
Hierarchical Decoupling and Edge Optimization
A final strategy involves separating the robot's cognitive reasoning from its motor execution. This is often framed as a hierarchical control problem: a slow, high-capacity model acts as a "planner" or "manager," while a fast, lightweight policy acts as the "worker."
┌────────────────────────────────────────────────────────┐
│ High-Level Cognitive Brain (VLM / LLM) │
│ Low Frequency (~1 Hz) | High Latency | Global Context │
└───────────────────────────┬────────────────────────────┘
│ (Subgoals / Constraints)
▼
┌────────────────────────────────────────────────────────┐
│ Low-Level Motor Controller (Adapters / Physics Model) │
│ High Frequency (100+ Hz) | Low Latency | Reactive │
└────────────────────────────────────────────────────────┘
Rongyu Zhang et al. implement this concept in MoLe-VLA by using a dynamic layer-skipping router. The model bypasses heavy language-processing layers when executing routine motor tasks, slashing computational overhead while preserving high-level reasoning when the scene changes.
Similarly, Zakariae El Asri addresses inference delays by pairing a slow model-based planner with a fast, physics-informed local controller. If the planner takes too long to compute a new path, the physics-informed controller fills the execution gap, ensuring the robot never lacks a valid command.
When deploying these models directly onto affordable edge hardware, engineers must also optimize the underlying computation graph. Haizhou Ge et al. demonstrate that combining input shape unification and symmetric quantization with an asynchronous execution pipeline allows complex Transformer-based policies to run smoothly on low-cost embedded platforms, eliminating the jerky, discontinuous movements caused by hardware-induced latency.