Asynchronous Robot Policies

Sombit Dey, Luc Van Gool, Danda Pani PaudelarXiv 2026 · arXiv

TL;DR

Asynchronous Robot Policies (ARP) decouple high-level semantic reasoning from low-level motor control, breaking the fundamental trade-off between model scale and robot agility. By running a massive Vision-Language Model (VLM) asynchronously to update a "semantic memory" while a lightweight expert generates actions at high frequency, ARP allows robots to be both "smart" and "fast," achieving up to 5x better performance in dynamic environments compared to standard sequential architectures.

What Problem Does This Paper Address?

In the current "scaling law" era, roboticists face a frustrating reactivity-capability trade-off. Large Vision-Language-Action (VLA) models are excellent at high-level reasoning and generalization, but their massive parameter counts introduce significant computational latency.

Standard VLA architectures are sequential: the robot observes, the VLM "thinks," and then the robot acts. If the VLM takes 200ms to process a frame, the control loop is stuck at 5Hz. This is far too slow for real-world robotics, where handling sudden bumps, grasping moving objects, or reacting to contact dynamics requires tight perception-action loops (often 20Hz or higher). Consequently, scaling the "brain" of the robot has historically made its "reflexes" sluggish and lag-bound.

Methods

The authors propose Asynchronous Robot Policy (ARP), which moves away from sequential execution toward a multi-rate, decoupled architecture.

Figure 1: a) Architectural and formulation differences between Asynchronous Robot Policies (ARP) and the standard Vision-Language-Action (VLA) paradigm. b) Performance comparison between ARP and standard VLAs across two key axes (both, higher the better): success rate and reactivity .

1. The Two-Loop Architecture

  • Slow Semantic Loop (Semantic Expert): A high-capacity VLM periodically processes observations to produce a compact, fixed-dimensional semantic memory. Instead of just using text outputs, ARP taps into the VLM's internal Key-Value (KV) cache activations to capture rich, hierarchical features.
  • Fast Action Loop (Vision-Action Expert): A lightweight, bidirectional Transformer operates at a high control frequency. It takes real-time visual and proprioceptive inputs and uses cross-attention to modulate its actions based on the most recently available semantic memory.

2. Handling "Staleness"

Because the loops are asynchronous, the semantic memory available to the fast loop is always slightly "stale" (outdated). To fix this, the authors introduce:

  • Explicit Conditioning: The action expert is explicitly conditioned on memory staleness (the time elapsed since the memory's observation was taken) and execution delay.
  • Stale-Cache Training: During training, the model is exposed to temporally mismatched observation-memory pairs. This forces the action expert to learn how to compensate for delays rather than assuming perfect synchronization.
Figure 2: Architecture design behind ARP. The semantic expert populates the memory with the latest available observation, whereas the vision- action expert cross attends to the stale-memory. The adaptive normalization is influenced by both the flow step τ as well as staleness.

Datasets

The researchers evaluated ARP across a diverse set of benchmarks:

  • LIBERO: A standard suite for multi-task robot manipulation.
  • LIBERO-Perturb (New): A diagnostic benchmark introduced in this paper where the target object is suddenly shifted by 5cm mid-episode to test real-time reactivity.
  • Dynamic Bump (New): A benchmark where the robot must grasp a moving object that receives a random velocity impulse (a "bump") during the task.
  • DROID: A large-scale, "in-the-wild" real-world dataset used for training generalist policies.
  • RoboLab: A high-fidelity, photorealistic simulation environment used to evaluate models trained on DROID.

Key Insights and Ideas

  • Decoupling is the "Aha!" Moment: You don't need the high-level reasoning engine to run at 50Hz. As long as the "reflexive" expert has a good, albeit slightly old, summary of the scene's semantics, it can handle high-frequency motor adjustments independently.
  • Staleness as a Feature, Not a Bug: By treating latency as an input variable ($d_t$), the model learns to "look ahead" or adjust its confidence based on how old its semantic information is.
  • Anticipatory Representations: The authors hypothesize that training with staleness forces the VLM to generate "temporally invariant, anticipatory representations"—essentially teaching the brain to provide a more robust "plan" that remains valid even as the world changes slightly.
  • Extreme Efficiency: ARP allows the VLM to run at 10x lower frequency than its nominal rate with virtually no performance loss. This significantly reduces the compute budget and energy consumption required for deployment.

Key Results

  • Superior Reactivity: In the Dynamic Bump task, ARP-1B achieved a 44.7% success rate, while the standard sequential $\pi_0$ baseline collapsed to just 12.7%.
  • Robustness to Perturbations: On the LIBERO-Perturb benchmark, ARP maintained a 51.5% success rate against sudden object shifts, compared to a meager 9.5% for the baseline.
  • Efficiency Gains: ARP achieved an 18% higher success rate while requiring 20% less time to complete tasks in quasi-static setups.
  • Real-World Performance: On physical Franka robot tasks (like "Vegetables in Bowl"), ARP was significantly more robust and efficient, showing higher throughput than synchronous counterparts.
Figure 4: Qualitative example of Dynamic-Bump task. After bumping the ball, ARP is able to react in time to grasp it whereas the π0 model fails.

Limitations and Future Work

While ARP provides a powerful blueprint for reactive robotics, some challenges remain:

  • Communication Overhead: The paper notes that optimizing inter-process communication (IPC) is critical, as the benefits of asynchrony can be eaten up by slow data transfer between the "brain" and "reflex" processes.
  • Long-Horizon Memory: While the current semantic memory works for immediate tasks, the authors suggest exploring more persistent memory structures for very long-horizon reasoning.
  • Full-Duplex Streams: Future work could involve more complex, two-way communication where the action expert also provides feedback to the semantic expert to guide its focus.

Summary updated

Abstract

Scaling vision-language foundation models has significantly advanced generalist robot manipulation, but the increased model scale introduces computational latency that degrades real-time physical reactivity. Existing Vision-Language-Action (VLA) architectures execute high-level semantic reasoning and action ex- ecution sequentially within the primary control loop, creating a fundamental tension between semantic-reasoning capability and robot agility. To address this, we propose Asynchronous Robot Policy (ARP), an architectural framework that decouples semantic reasoning from low-level action generation using asynchronous processes operating at independent frequencies. In ARP, a high-capacity Vision-Language Model (VLM) asynchronously updates a semantic memory at low fre- quency, while a lightweight Vision-Action Expert, conditioned on this semantic memory, generates robot actions at a much higher control frequency. Such a formulation, enables having a fast perception-action loop without compromising on the semantic capability. To overcome asynchronous memory staleness, ARP trains the vision-action expert on the semantic staleness and execution delay, with temporally mismatched observation across processes. On Franka manipulation tasks, ARP achieves an 18% higher success rate and 20% less time need for execution in quasi-static setups, while 3-5 times baseline performance in dynamic simulation environments. Furthermore, ARP enables operating the VLM at 10 times lower than its nominal frequency with no significant performance loss making it much more efficient than the baseline. These results highlight the architectural advantages of modeling robot policies as asynchronous processes, enabling real-world reactive deployment without sacrificing the scale of foundation models.
BibTeX
@misc{dey2026asynchronous,
  author = {Sombit Dey and Luc Van Gool and Danda Pani Paudel},
  title = {{Asynchronous Robot Policies}},
  year = {2026},
  eprint = {2611.05155},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05155},
}
Figure