TL;DR
Retrieve to Act (R2A) is a test-time scaling framework that significantly improves the robustness of Vision-Language-Action (VLA) models against visual perturbations by retrieving successful motion patterns from a geometric "experience graph." By harmonizing fragile visual predictions with stable kinematic priors, R2A achieves up to a 40% success rate improvement in real-world deployments without requiring any finetuning of the base policy.
What Problem Does This Paper Address?
Modern Vision-Language-Action (VLA) models like OpenVLA or $\pi_{0.5}$ are remarkably good at understanding semantic instructions ("pick up the banana"). However, they suffer from extreme kinematic fragility. Because these models rely heavily on visual features, even minor "out-of-distribution" (OOD) shifts—such as a different camera angle, a new background texture, or changed lighting—can cause the robot's trajectory to diverge wildly, leading to catastrophic failure.
The authors argue that while the visuals might change, the underlying kinematics required to solve a task often remain the same. A robot's own successful training demonstrations contain a wealth of uncorrupted geometric priors. The challenge lies in how to effectively retrieve and apply this prior experience during execution when the current visual input is noisy or confusing.
Methods
R2A introduces a three-stage pipeline that acts as a "safety net" for frozen VLA policies:
- Motion Primitive Graph Construction: Offline, the system organizes successful training demonstrations into a spatiotemporal graph. Nodes represent pure kinematic states (joint angles, poses), and edges represent either temporal transitions (what happens next) or spatial proximity (similar poses across different tasks). Crucially, this graph ignores pixels entirely.
- Initial State Rectification: If a robot starts in an OOD pose (e.g., the arm was bumped before starting), a diffusion-based module projects the initial state back onto the "training manifold," ensuring the execution starts from a known-good configuration.
- Query-Dependent GNN Retrieval: During execution, R2A uses a Graph Neural Network (GNN) to "localize" the robot's current progress within the motion graph. It takes the VLA's current prediction and the robot's recent history as a query to find the most compatible future motion primitives.
- Action Harmonization: A learned, dimension-wise gating mechanism fuses the VLA's predicted action with the retrieved geometric prior. If the VLA's prediction deviates significantly from the successful patterns in the graph, the gate adaptively relies more on the retrieved motion to rectify the trajectory.
Datasets
The authors evaluated R2A across several challenging benchmarks:
- LIBERO-Plus: A simulation suite featuring 7 types of systematic perturbations (camera, lighting, layout, etc.).
- RoboTwin2.0-Plus: A benchmark for bimanual (two-armed) manipulation tasks.
- Real-World Deployment: Tasks performed on a Mobile ALOHA platform, specifically "Banana2Pot" (picking and placing) and "Cube2Drawer."
Key Insights and Ideas
- Geometry as an Anchor: By stripping away visual features and focusing on kinematics, R2A creates a representation that is inherently immune to visual noise.
- Retrieval as Localization: The GNN doesn't just look for a "match"; it treats the problem as localizing the robot's state in a topological map of success. This allows it to suggest "what comes next" based on the structure of the task.
- Selective Intervention: The harmonization gate is key. It doesn't blindly overwrite the VLA; it preserves the VLA's local reactivity (its ability to respond to small, real-time changes) while providing a "global" correction if the VLA starts to drift off-course.
- Test-Time Scaling: Unlike methods that require expensive finetuning on augmented data, R2A is a plug-and-play module that can be added to any existing VLA.
Key Results
R2A consistently outperformed base policies and other state-of-the-art OOD mitigation methods:
- Simulation Gains: On LIBERO-Plus, R2A improved the average success rate of GR00T N1.7 from 77.1% to 89.0% and StableVLA from 78.1% to 89.5%.
- Real-World Impact: In physical deployments with the Aloha robot, R2A achieved up to a 40% absolute improvement in success rates under perturbed initial states.
- Efficiency: R2A adds only a modest latency overhead (~21ms per step), making it significantly faster than methods that rely on external Vision-Language Models (VLMs) for steering, which can take over 800ms per step.
Limitations and Future Work
While powerful, R2A has some notable limitations:
- Lack of Scene Geometry: The retrieval index doesn't explicitly account for new obstacles in the scene. It relies on the VLA to "see" them, which might fail if the VLA is already confused.
- Closed-World Retrieval: The system can only retrieve motion patterns that already exist in the demonstration graph. It cannot autonomously invent novel corrective sequences for entirely new failure modes.
- Future Directions: The authors suggest integrating R2A with multi-view visual inputs to better ground motions in 3D space and exploring object-centric memory to handle complex, multi-stage tasks more effectively.
Summary updated