Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement

Kinam Kim, Namiko Saito, Heecheol Kim, Katsushi Ikeuchi, Jaegul Choo, Yasuyuki MatsushitaarXiv 2026 · arXiv

TL;DR

This paper introduces an object-centric residual reinforcement learning (RL) framework that enhances the precision and robustness of Vision-Language-Action (VLA) robot models zero-shot from simulation to reality. By operating on a low-dimensional, domain-invariant observation space—specifically 6-DoF object poses, proprioception, and base VLA actions—the method sidesteps the traditional visual sim-to-real gap. Deployed on a real Franka Research 3 robot across five manipulation tasks, it boosts the average success rate from 42% to 76% without any real-world training.

What Problem Does This Paper Address?

Vision-Language-Action (VLA) models have advanced the field of robotic manipulation by enabling generalist robots to follow diverse language instructions. However, because these models are typically trained via imitation learning, they suffer from compounding execution errors. A tiny mistake early in a trajectory can lead to an unfamiliar state, causing the robot to fail entirely during precise physical interactions (such as grasping a small block or inserting an object).

Reinforcement learning (RL) is a natural fit for learning corrective behaviors, but training RL agents directly in the real world is slow, costly, and potentially unsafe. Conversely, training corrective "residual" policies in simulation and transferring them to reality introduces a classic sim-to-real dilemma:

  1. Privileged-state methods rely on simulator-only data (like exact contact forces) and require lossy distillation to deploy on real images.
  2. Image-based methods suffer from massive visual domain gaps between simulated renderings and real-world camera feeds.

The authors tackle this challenge by asking: Can we train a lightweight corrective policy purely in simulation that transfers zero-shot to a real robot to fix VLA errors?

Figure 1: Object-centric residual RL for zero-shot sim-to-real VLA enhancement. The base VLA fails on the real robot (left). A residual policy trained purely in simulation (middle) is added zero-shot to recover task success on the same real-robot setup (right).

Methods

The proposed framework, Object-Centric Residual RL, breaks the enhancement pipeline into three distinct stages:

  1. Paired Sim/Real VLA via Teleoperation Replay: To ensure the simulation and real-world action distributions match, the authors replay the real-world teleoperated demonstrations inside a MuJoCo simulator. They train a "sim VLA" alongside the "real VLA." Because both models are supervised by identical action sequences, their outputs remain closely aligned despite different visual inputs.
  2. Object-Centric Residual RL Training: A residual policy ($\pi_{res}^{sim}$) is trained in simulation using Twin Delayed DDPG (TD3). Crucially, it does not look at images. Instead, its observation space consists of domain-invariant features: the 6-DoF poses of task-relevant objects, the robot's proprioceptive state, and the current action chunk predicted by the frozen sim VLA. To make the policy robust to real-world perception errors, the authors inject structured noise into the object poses and implement a random "pose dropout" strategy during training.
  3. Zero-Shot Real Deployment: During real-world deployment, both the real VLA and the sim-trained residual policy are frozen. An off-the-shelf pose estimator (FoundationPose + SAM2) tracks the 6-DoF object poses in real time. If the tracking confidence drops below a certain threshold, a confidence-gated fallback mechanism zeroes out the pose vector, triggering the fallback strategy learned during the training dropout phase. The final action sent to the robot is a composition of the base VLA action and the residual correction.
Figure 2: Overview of the object-centric residual RL pipeline.

VLA Self-Improvement Loop

Beyond deployment, the framework features an autonomous self-improvement loop. Successful real-world rollouts corrected by the residual policy can be collected and used to retrain the base VLA via supervised fine-tuning (SFT), improving the generalist model without requiring additional human teleoperation.

Datasets

The method was evaluated across five distinct tabletop manipulation tasks:

  • Cube Lift: Grasping a cube and lifting it 3 cm.
  • Pick-and-Place: Picking a cube and placing it into a bowl.
  • Stack Cube: Picking a white cube and stacking it on a green cube.
  • Close Drawer: Pushing an open cabinet drawer closed.
  • Stand Cup Up: Grasping a cup lying on its side and standing it upright.

For each task, the base VLA models (GR00T-N1.5 and $\pi_{0.5}$) were fine-tuned on 30 teleoperation demonstrations per domain (sim and real).

Key Insights and Ideas

  • Sidestepping the Visual Gap: Instead of trying to bridge the visual sim-to-real gap using complex domain randomization or image-to-image translation, the authors completely remove images from the residual policy's observation space. 6-DoF object poses serve as a compact, domain-invariant bridge.
  • Behavioral Consistency: Because the sim VLA and real VLA are trained on identical teleoperation data, they fail in the exact same ways (e.g., hovering just above an object or stopping short of a target). This allows the residual policy to learn fixes for specific failure modes in simulation that translate perfectly to the real robot.
  • Emergent Strategies: Through RL exploration in simulation, the residual policy discovers clever corrective behaviors never seen in the human demonstrations, such as nudging/pre-rotating a cube into a better orientation before attempting a grasp.

Key Results

  • Substantial Success Rate Boost: Across the five real-world tasks on a Franka Research 3 robot, the framework improved the average success rate from 42% (base VLA) to 76% (with residual RL) zero-shot.
  • Increased Efficiency: The residual corrections steered the robot more directly toward its goals, resulting in a 9% to 22% reduction in task completion time (episode length).
  • Architecture Agnostic: The framework successfully enhanced multiple VLA backbones, providing similar performance boosts for both GR00T-N1.5 and $\pi_{0.5}$.
  • Data Bootstrapping: Retraining the base VLA on the residual-corrected rollouts via SFT successfully bumped the standalone VLA success rate from 42% to 59% while shortening its trajectories.

Limitations and Future Work

  • Perception Dependencies: The system relies heavily on real-time 6-DoF pose tracking. If an object is fully occluded by the gripper or hidden in heavy clutter, the pose estimator can fail or pass drifted estimates through the confidence gate, leading to incorrect residual movements.
  • The Dynamics Gap: While the object-centric representation bypasses the visual gap, it does not account for differences in physics (e.g., contact friction, gripper compliance). This can lead to suboptimal corrections in highly contact-rich tasks.
  • Manual Specification: Task-relevant objects must currently be specified manually. The authors note that scaling this to open-world settings will require automatic object identification, potentially leveraging VLA attention maps.

Summary updated

Abstract

Vision-Language-Action (VLA) models can generalize across diverse manipulation tasks, but their imitation-learning-based policies remain brittle in precise physical interactions due to compounding execution errors; Can a reinforcement learning policy trained purely in simulation improve the robustness of real-world VLAs zero-shot? Residual RL, which learns a corrective policy on top of a frozen VLA, offers a natural framework, but existing approaches face a fundamental sim-to-real dilemma: privileged-state methods require lossy distillation for deployment; image-based methods suffer from the visual domain gap; and real-world RL is costly and unsafe. We propose an object-centric residual RL framework that refines VLA actions using object poses, enabling a compact observation space that transfers consistently between simulation and reality. To align the two domains, we additionally replay the same teleoperation demonstrations in simulation to train a sim counterpart of the real-world VLA. The residual RL policy is trained only in simulation with pose noise injection and dropout, and transfers zero-shot to the real robot. Across five manipulation tasks on a real Franka Research 3 (FR3) robot, our method improves the success rate from 42% to 76% zero-shot, and the improved rollouts can be further reused to retrain the base VLA for self-improvement without additional teleoperation. Project page: https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/
BibTeX
@misc{kim2026object,
  author = {Kinam Kim and Namiko Saito and Heecheol Kim and Katsushi Ikeuchi and Jaegul Choo and Yasuyuki Matsushita},
  title = {{Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement}},
  year = {2026},
  eprint = {2606.18953},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.18953},
}

Loading related content...

Figure