Retrieve to Act: Motion Primitive Graph Retrieval for Robust VLA Execution

Pham Tri Quang, Bao-Ngoc Dao, Minh Duc Nguyen, Duc-Duy Nguyen, Hung T. Ho, Linh Dang Le, Trong-Bao Ho, Thien-Loc Ha, Tan Q. Nguyen, Tuan Dam, Vu N. Duong, Daniel Sonntag, Ngan Le, Khoa D. Doan, Jan Peters, An Thai Le, Duy M. H. Nguyen, Vien Anh NgoarXiv 2026 · arXiv

TL;DR

Retrieve to Act (R2A) is a test-time scaling framework that significantly improves the robustness of Vision-Language-Action (VLA) models against visual perturbations by retrieving successful motion patterns from a geometric "experience graph." By harmonizing fragile visual predictions with stable kinematic priors, R2A achieves up to a 40% success rate improvement in real-world deployments without requiring any finetuning of the base policy.

What Problem Does This Paper Address?

Modern Vision-Language-Action (VLA) models like OpenVLA or $\pi_{0.5}$ are remarkably good at understanding semantic instructions ("pick up the banana"). However, they suffer from extreme kinematic fragility. Because these models rely heavily on visual features, even minor "out-of-distribution" (OOD) shifts—such as a different camera angle, a new background texture, or changed lighting—can cause the robot's trajectory to diverge wildly, leading to catastrophic failure.

The authors argue that while the visuals might change, the underlying kinematics required to solve a task often remain the same. A robot's own successful training demonstrations contain a wealth of uncorrupted geometric priors. The challenge lies in how to effectively retrieve and apply this prior experience during execution when the current visual input is noisy or confusing.

Methods

R2A introduces a three-stage pipeline that acts as a "safety net" for frozen VLA policies:

  1. Motion Primitive Graph Construction: Offline, the system organizes successful training demonstrations into a spatiotemporal graph. Nodes represent pure kinematic states (joint angles, poses), and edges represent either temporal transitions (what happens next) or spatial proximity (similar poses across different tasks). Crucially, this graph ignores pixels entirely.
  2. Initial State Rectification: If a robot starts in an OOD pose (e.g., the arm was bumped before starting), a diffusion-based module projects the initial state back onto the "training manifold," ensuring the execution starts from a known-good configuration.
  3. Query-Dependent GNN Retrieval: During execution, R2A uses a Graph Neural Network (GNN) to "localize" the robot's current progress within the motion graph. It takes the VLA's current prediction and the robot's recent history as a query to find the most compatible future motion primitives.
  4. Action Harmonization: A learned, dimension-wise gating mechanism fuses the VLA's predicted action with the retrieved geometric prior. If the VLA's prediction deviates significantly from the successful patterns in the graph, the gate adaptively relies more on the retrieved motion to rectify the trajectory.
The R2A pipeline: Offline graph construction followed by test-time rectification, retrieval, and harmonization.

Datasets

The authors evaluated R2A across several challenging benchmarks:

  • LIBERO-Plus: A simulation suite featuring 7 types of systematic perturbations (camera, lighting, layout, etc.).
  • RoboTwin2.0-Plus: A benchmark for bimanual (two-armed) manipulation tasks.
  • Real-World Deployment: Tasks performed on a Mobile ALOHA platform, specifically "Banana2Pot" (picking and placing) and "Cube2Drawer."

Key Insights and Ideas

  • Geometry as an Anchor: By stripping away visual features and focusing on kinematics, R2A creates a representation that is inherently immune to visual noise.
  • Retrieval as Localization: The GNN doesn't just look for a "match"; it treats the problem as localizing the robot's state in a topological map of success. This allows it to suggest "what comes next" based on the structure of the task.
  • Selective Intervention: The harmonization gate is key. It doesn't blindly overwrite the VLA; it preserves the VLA's local reactivity (its ability to respond to small, real-time changes) while providing a "global" correction if the VLA starts to drift off-course.
  • Test-Time Scaling: Unlike methods that require expensive finetuning on augmented data, R2A is a plug-and-play module that can be added to any existing VLA.

Key Results

R2A consistently outperformed base policies and other state-of-the-art OOD mitigation methods:

  • Simulation Gains: On LIBERO-Plus, R2A improved the average success rate of GR00T N1.7 from 77.1% to 89.0% and StableVLA from 78.1% to 89.5%.
  • Real-World Impact: In physical deployments with the Aloha robot, R2A achieved up to a 40% absolute improvement in success rates under perturbed initial states.
  • Efficiency: R2A adds only a modest latency overhead (~21ms per step), making it significantly faster than methods that rely on external Vision-Language Models (VLMs) for steering, which can take over 800ms per step.
Figure 3: R2A under Sensor Noise (GR00T N1.7). The query retrieves a demonstration go- ing down to the book, and the executed action blends it with the policy. Background shown from a clean frame for readability.

Limitations and Future Work

While powerful, R2A has some notable limitations:

  • Lack of Scene Geometry: The retrieval index doesn't explicitly account for new obstacles in the scene. It relies on the VLA to "see" them, which might fail if the VLA is already confused.
  • Closed-World Retrieval: The system can only retrieve motion patterns that already exist in the demonstration graph. It cannot autonomously invent novel corrective sequences for entirely new failure modes.
  • Future Directions: The authors suggest integrating R2A with multi-view visual inputs to better ground motions in 3D space and exploring object-centric memory to handle complex, multi-stage tasks more effectively.

Summary updated

Abstract

Vision-Language-Action (VLA) policies demonstrate remarkable semantic generalization, yet remain highly vulnerable to out-of-distribution (OOD) visual shifts during execution. While existing test-time methods attempt to suppress spurious visual cues, they fundamentally overlook the rich, uncorrupted kinematic priors embedded in the policy's own training experience. We introduce Retrieve to Act (R2A), a test-time scaling framework that formulates OOD robustness purely as a geometric motion primitive retrieval problem. R2A bypasses fragile visual features by structuring successful training demonstrations into an offline spatiotemporal motion primitive graph. During inference, a query-dependent Graph Neural Network dynamically localizes the robot's ongoing execution within this graph, retrieving structurally compatible future motion primitives. These retrieved geometric priors are then seamlessly harmonized with the frozen VLA's predicted actions via a learned, dimension-wise gating mechanism, adaptively rectifying erroneous spatial trajectories while preserving the policy's local reactivity. Extensive experiments across three state-of-the-art VLA architectures, GR00T N1.7, $π_{0.5}$, and StableVLA, demonstrate that R2A consistently and significantly enhances robustness against diverse contextual perturbations. R2A achieves absolute success rate improvements of up to +12% in simulation benchmarks (LIBERO-Plus, RoboTwin2.0-Plus) and +40% in real-world deployments, without requiring any finetuning of the base policies. Project page: https://retrieve2act.github.io/r2a/
BibTeX
@misc{quang2026retrieve,
  author = {Pham Tri Quang and Bao-Ngoc Dao and Minh Duc Nguyen and Duc-Duy Nguyen and Hung T. Ho and Linh Dang Le and Trong-Bao Ho and Thien-Loc Ha and Tan Q. Nguyen and Tuan Dam and Vu N. Duong and Daniel Sonntag and Ngan Le and Khoa D. Doan and Jan Peters and An Thai Le and Duy M. H. Nguyen and Vien Anh Ngo},
  title = {{Retrieve to Act: Motion Primitive Graph Retrieval for Robust VLA Execution}},
  year = {2026},
  eprint = {2610.37215},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37215},
}
Figure