TL;DR
This paper introduces an object-centric residual reinforcement learning (RL) framework that enhances the precision and robustness of Vision-Language-Action (VLA) robot models zero-shot from simulation to reality. By operating on a low-dimensional, domain-invariant observation space—specifically 6-DoF object poses, proprioception, and base VLA actions—the method sidesteps the traditional visual sim-to-real gap. Deployed on a real Franka Research 3 robot across five manipulation tasks, it boosts the average success rate from 42% to 76% without any real-world training.
What Problem Does This Paper Address?
Vision-Language-Action (VLA) models have advanced the field of robotic manipulation by enabling generalist robots to follow diverse language instructions. However, because these models are typically trained via imitation learning, they suffer from compounding execution errors. A tiny mistake early in a trajectory can lead to an unfamiliar state, causing the robot to fail entirely during precise physical interactions (such as grasping a small block or inserting an object).
Reinforcement learning (RL) is a natural fit for learning corrective behaviors, but training RL agents directly in the real world is slow, costly, and potentially unsafe. Conversely, training corrective "residual" policies in simulation and transferring them to reality introduces a classic sim-to-real dilemma:
- Privileged-state methods rely on simulator-only data (like exact contact forces) and require lossy distillation to deploy on real images.
- Image-based methods suffer from massive visual domain gaps between simulated renderings and real-world camera feeds.
The authors tackle this challenge by asking: Can we train a lightweight corrective policy purely in simulation that transfers zero-shot to a real robot to fix VLA errors?
Methods
The proposed framework, Object-Centric Residual RL, breaks the enhancement pipeline into three distinct stages:
- Paired Sim/Real VLA via Teleoperation Replay: To ensure the simulation and real-world action distributions match, the authors replay the real-world teleoperated demonstrations inside a MuJoCo simulator. They train a "sim VLA" alongside the "real VLA." Because both models are supervised by identical action sequences, their outputs remain closely aligned despite different visual inputs.
- Object-Centric Residual RL Training: A residual policy ($\pi_{res}^{sim}$) is trained in simulation using Twin Delayed DDPG (TD3). Crucially, it does not look at images. Instead, its observation space consists of domain-invariant features: the 6-DoF poses of task-relevant objects, the robot's proprioceptive state, and the current action chunk predicted by the frozen sim VLA. To make the policy robust to real-world perception errors, the authors inject structured noise into the object poses and implement a random "pose dropout" strategy during training.
- Zero-Shot Real Deployment: During real-world deployment, both the real VLA and the sim-trained residual policy are frozen. An off-the-shelf pose estimator (FoundationPose + SAM2) tracks the 6-DoF object poses in real time. If the tracking confidence drops below a certain threshold, a confidence-gated fallback mechanism zeroes out the pose vector, triggering the fallback strategy learned during the training dropout phase. The final action sent to the robot is a composition of the base VLA action and the residual correction.
VLA Self-Improvement Loop
Beyond deployment, the framework features an autonomous self-improvement loop. Successful real-world rollouts corrected by the residual policy can be collected and used to retrain the base VLA via supervised fine-tuning (SFT), improving the generalist model without requiring additional human teleoperation.
Datasets
The method was evaluated across five distinct tabletop manipulation tasks:
- Cube Lift: Grasping a cube and lifting it 3 cm.
- Pick-and-Place: Picking a cube and placing it into a bowl.
- Stack Cube: Picking a white cube and stacking it on a green cube.
- Close Drawer: Pushing an open cabinet drawer closed.
- Stand Cup Up: Grasping a cup lying on its side and standing it upright.
For each task, the base VLA models (GR00T-N1.5 and $\pi_{0.5}$) were fine-tuned on 30 teleoperation demonstrations per domain (sim and real).
Key Insights and Ideas
- Sidestepping the Visual Gap: Instead of trying to bridge the visual sim-to-real gap using complex domain randomization or image-to-image translation, the authors completely remove images from the residual policy's observation space. 6-DoF object poses serve as a compact, domain-invariant bridge.
- Behavioral Consistency: Because the sim VLA and real VLA are trained on identical teleoperation data, they fail in the exact same ways (e.g., hovering just above an object or stopping short of a target). This allows the residual policy to learn fixes for specific failure modes in simulation that translate perfectly to the real robot.
- Emergent Strategies: Through RL exploration in simulation, the residual policy discovers clever corrective behaviors never seen in the human demonstrations, such as nudging/pre-rotating a cube into a better orientation before attempting a grasp.
Key Results
- Substantial Success Rate Boost: Across the five real-world tasks on a Franka Research 3 robot, the framework improved the average success rate from 42% (base VLA) to 76% (with residual RL) zero-shot.
- Increased Efficiency: The residual corrections steered the robot more directly toward its goals, resulting in a 9% to 22% reduction in task completion time (episode length).
- Architecture Agnostic: The framework successfully enhanced multiple VLA backbones, providing similar performance boosts for both GR00T-N1.5 and $\pi_{0.5}$.
- Data Bootstrapping: Retraining the base VLA on the residual-corrected rollouts via SFT successfully bumped the standalone VLA success rate from 42% to 59% while shortening its trajectories.
Limitations and Future Work
- Perception Dependencies: The system relies heavily on real-time 6-DoF pose tracking. If an object is fully occluded by the gripper or hidden in heavy clutter, the pose estimator can fail or pass drifted estimates through the confidence gate, leading to incorrect residual movements.
- The Dynamics Gap: While the object-centric representation bypasses the visual gap, it does not account for differences in physics (e.g., contact friction, gripper compliance). This can lead to suboptimal corrections in highly contact-rich tasks.
- Manual Specification: Task-relevant objects must currently be specified manually. The authors note that scaling this to open-world settings will require automatic object identification, potentially leveraging VLA attention maps.
Summary updated