Zero-Shot VLA Adaptation with Retrieved Guidance

Subin Kim, Anh-Quan Pham, Dinesh JayaramanarXiv 2026 · arXiv

TL;DR

RAGu (Retrieval-Augmented Guidance) brings the power of Retrieval-Augmented Generation to robotics, allowing frozen Vision-Language-Action (VLA) models to adapt to new environments and robots without any retraining. By "looking up" relevant demonstrations from external datasets, warping them to fit the current scene, and using them to steer the model's output at inference time, RAGu boosts real-world success rates from 20% to 80%.

What Problem Does This Paper Address?

Large-scale Vision-Language-Action (VLA) models are the "foundation models" of robotics, but they are notoriously brittle. Even minor changes—a different camera angle, a new background, or a slightly different robot arm—can cause these models to fail.

In the world of Large Language Models (LLMs), we solve similar "knowledge gaps" using Retrieval-Augmented Generation (RAG), where the model looks up relevant documents to help answer a query. However, applying RAG to robotics is hard because:

  1. Domain Mismatch: A demonstration of a robot picking up a towel in a lab doesn't look like a robot picking up a towel in a home kitchen.
  2. Policy Incompatibility: A retrieved trajectory might be geometrically close but kinematically impossible or awkward for the specific VLA to follow.
  3. Context Limits: Unlike text, you can't just "paste" several video demonstrations into a VLA's limited input window.

Methods

The authors introduce RAGu, a training-free framework that adapts a pre-trained VLA to a new domain using a three-step pipeline: Warp $\rightarrow$ Retrieve $\rightarrow$ Steer.

Overview of the RAGu pipeline showing semantic filtering, trajectory warping, policy-aware retrieval, and inference steering.
  1. Semantic Warping: RAGu filters a large, "in-the-wild" dataset (like DROID) for relevant tasks. It then uses semantic bounding boxes (via Grounding DINO) to identify key objects in both the source demonstration and the target scene. It "warps" the source trajectory to fit the new spatial layout, turning mismatched data into usable spatial priors.
  2. Policy-Aware Retrieval: Instead of just picking the "closest" trajectory, RAGu uses a coarse-to-fine strategy. It first prunes candidates using Dynamic Time Warping (DTW). Then, it selects the trajectory that is most compatible with the VLA's internal "intent" by measuring the gradient norm of the guidance cost—essentially asking, "Which path requires the least effort for the VLA to follow?"
  3. Inference-Time Steering: Rather than retraining the model, RAGu uses the retrieved trajectory to define an energy field. This field "pulls" the VLA's generative process (specifically its flow-matching denoising steps) toward the desired path while still allowing the VLA to handle the low-level closed-loop control.

Datasets

The researchers evaluated RAGu across several challenging transfers:

  • DROID: A large-scale real-world dataset featuring a 7-DoF Franka arm, used as the primary source of "in-the-wild" demonstrations.
  • MetaWorld: A simulation dataset (Sawyer arm) used to test sim-to-real transfer.
  • RoboLab: A high-fidelity simulation benchmark used for systematic evaluation.
  • RoboCasa: A simulation environment used to test cross-morphology transfer to a 29-DoF GR-1 humanoid robot.

Key Insights and Ideas

  • Compatibility Over Proximity: A key "aha" moment was finding that geometric similarity isn't enough. By selecting trajectories based on the VLA's own policy gradients, the system finds paths that the robot can execute more smoothly and reliably.
  • Cartesian Traces as a Universal Language: By performing guidance in 3D Cartesian space (the path of the wrist) rather than joint space, RAGu can transfer knowledge between completely different robots—for example, using data from a single Franka arm to guide a complex bimanual humanoid.
  • Zero-Shot Adaptation: RAGu requires zero new training data and zero weight updates. It effectively "unlocks" the latent capabilities of a frozen VLA by providing it with a better "map" for the specific task at hand.

Key Results

RAGu demonstrated significant performance gains across the board:

  • Real-World Success: On a suite of 5 diverse tasks (like opening drawers and wiping tables), RAGu improved the success rate of a base VLA from 20% to 80%.
  • Humanoid Manipulation: It successfully guided a 29-DoF humanoid robot using data from a simple 7-DoF arm, increasing success rates by up to 13.6%.
  • Sim-to-Real: RAGu enabled a real Franka robot to perform a multi-stage "Banana-to-Plate" task using only simulated demonstrations from a different robot type (Sawyer).
Comparison of success rates in the real world across different tasks.

Qualitatively, RAGu produces much smoother and more task-appropriate behavior than baselines that rely on sparse waypoints, which often cause the robot to break contact with surfaces prematurely.

Rollout comparison on a wiping task showing RAGu's superior contact maintenance.

Limitations and Future Work

  • Database Quality: RAGu assumes the source database contains at least some demonstrations that approximately match the target task category.
  • Perception Sensitivity: The system relies on semantic correspondence. If the object detection or 3D triangulation fails (e.g., due to heavy occlusion), the resulting guidance signal will be corrupted.
  • Prior Constraints: The authors note that guidance can only reshape motion within the VLA's existing capabilities; it cannot magically enable a robot to perform a skill it has never fundamentally learned during pre-training.

Future work may involve improving the robustness of the semantic correspondence stage and exploring how to scale the retrieval pool even further without increasing latency.

Summary updated

Abstract

Pre-training large parametric "Vision-Language-Action" (VLA) models on large demonstration datasets has recently yielded promising results for general-purpose manipulation. However, these models still struggle under even small domain shifts relative to the training data, such as changes in camera viewpoint, background, object identity, or robot morphology. Motivated by the success of Retrieval-Augmented Generation (RAG) in language models, we pursue retrieval to boost VLA performance. Specifically, we seek to leverage retrieval from domain-mismatched, "in-the-wild" demonstration datasets like DROID to guide a VLA's behavior in a new target domain. To achieve this, we introduce RAGu, which overcomes robotics-specific challenges for RAG: (1) It incorporates simple geometric calibration to transform source demonstrations into the target setup and reduce cross-domain differences. (2) It uses a coarse-to-fine retrieval strategy that selects, from a large pool of pre-warped candidates, the trajectory most compatible with the base VLA. (3) It guides the VLA's flow-matching inference procedure rather than seeking to retrain the base model or expand its limited context window. We evaluate RAGu across real-world, simulation, sim-to-real transfer, and cross-embodiment humanoid manipulation. We find it improves real-world VLA success rates from 20% to 80% and outperforms the closest-related baselines in sim. Project website: raguvla.github.io
BibTeX
@misc{kim2026zero,
  author = {Subin Kim and Anh-Quan Pham and Dinesh Jayaraman},
  title = {{Zero-Shot VLA Adaptation with Retrieved Guidance}},
  year = {2026},
  eprint = {2611.10894},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10894},
}
Figure