Residual Reinforcement Learning for Visual Servo Control with Zero-Shot Sim-to-Real Transfer

Mahdi Nobar, Alisa Rupenyan, John Lygeros, Jürg KellerarXiv 2026 · arXiv

This paper presents a hybrid control strategy that combines classical control theory with deep reinforcement learning to achieve high-precision tracking of moving objects. By augmenting a standard industrial controller with a "residual" policy trained in simulation, the authors demonstrate sub-millimeter tracking accuracy on a real robot arm—surpassing the theoretical performance limits of traditional model-based approaches.

TL;DR

The researchers developed a "Residual SAC-PI" controller that combines a classical inverse-Jacobian PI controller with a Soft Actor-Critic (SAC) reinforcement learning agent. This hybrid approach, paired with a delay-compensated Kalman filter, allows a Franka Emika Panda robot to track conveyor-belt objects with sub-millimeter precision, successfully transferring a policy learned in a "digital twin" to a real robot with zero-shot fine-tuning.

What Problem Does This Paper Address?

In industrial automation, tracking an object on a moving conveyor belt is harder than it looks. Robots face three major hurdles:

  1. Kinematic Mismatch: The mathematical model of the robot (the "digital twin") is never a perfect match for the physical hardware. Small errors in link lengths or joint alignments lead to "kinematic bias."
  2. Sensor Latency: Vision systems (like stereo cameras) introduce significant delays. By the time a computer processes an image, the object has already moved.
  3. The Sim-to-Real Gap: Pure reinforcement learning (RL) agents often fail when moved from a clean simulation to a noisy real-world environment.

Traditional controllers (like PI controllers) are stable but struggle to compensate for these complex, unmodeled errors. Pure RL is adaptive but data-hungry and potentially unsafe on expensive hardware.

Methods

The authors propose a three-part framework:

  • Residual Learning Architecture: Instead of letting an RL agent control the robot directly, they use a nominal inverse-Jacobian PI controller to handle the bulk of the movement. A Soft Actor-Critic (SAC) agent then provides a "residual" command—a small correction added to the PI signal to cancel out errors the classical controller can't see.
  • Digital Twin with Domain Randomization: The SAC agent is trained entirely in a simulation (PyBullet). To ensure it works on the real robot, the authors used domain randomization, intentionally perturbing link positions, orientations, and sensor noise during training so the agent learns to be robust to uncertainty.
  • Delay-Compensated Kalman Filter (KF): To handle camera lag, they implemented a KF that "rewinds" the state estimate when a delayed measurement arrives, applies the correction at the correct historical timestamp, and then "fast-forwards" to the present to provide a latency-aware target estimate.
  • Frequency-Domain Analysis: The authors derived a theoretical lower bound for tracking error based on closed-loop sensitivity. This provided a benchmark to prove that their RL agent wasn't just "tuning" the controller, but actually outperforming the theoretical limits of the model-based class.

Datasets

The study did not use a standard public dataset. Instead, it utilized:

  • Simulation Environment: A custom digital twin of the Franka Emika Panda robot and a conveyor system built in PyBullet.
  • Real-World Validation: Experiments were conducted on a physical Franka Emika Panda robot cell equipped with a stereo camera and a stepper-motor-driven conveyor belt.

Key Insights and Ideas

  • The "Residual" Advantage: By only learning corrections, the RL agent benefits from the "prior knowledge" of the PI controller. This makes training more sample-efficient and keeps the robot's behavior grounded in stable control theory.
  • Beating the Theoretical Floor: The most striking insight is that the residual policy drove tracking error below the derived lower bound for classical model-based controllers. This suggests the RL agent successfully compensated for non-linearities and biases that are mathematically impossible for a standard PI controller to resolve.
  • Zero-Shot is Viable: The paper proves that with sufficient domain randomization (e.g., adding $\pm 0.5$ mm and $\pm 3^\circ$ offsets to joint frames in sim), an agent can move from simulation to reality without a single second of training on the physical hardware.

Key Results

  • Accuracy Boost: Across conveyor speeds ranging from 27.9 to 41.8 mm/s, the proposed method reduced mean tracking errors from 1.1–1.4 mm (baseline PI) to 0.3–0.7 mm.
  • Robustness: The residual SAC-PI controller outperformed the baseline even when the baseline was given the "true" (unbiased) kinematic model, proving it compensates for more than just link-length errors (e.g., it likely handles belt flexibility and unmodeled joint dynamics).
  • Spectral Performance: Frequency-domain plots showed that the SAC agent specifically reduced error power at low frequencies where the kinematic bias typically dominates.

Limitations and Future Work

  • Directional Analysis: The current study treats tracking error as a scalar magnitude. The authors note that future work should analyze directional effects—how modeling mismatches affect the X, Y, and Z axes differently.
  • Adaptive Mismatch Learning: While domain randomization works for zero-shot transfer, the authors suggest developing "adaptive mismatch learning" to allow the robot to identify and adapt to its specific hardware biases in real-time.
  • Task Complexity: The current work focuses on 1D tracking along a conveyor. Extending this to complex 3D trajectories or dynamic obstacle avoidance would be a logical next step.

Summary updated

Abstract

We address tracking a moving object on a conveyor with a Franka Emika Panda robot arm, despite an imprecise digital twin, measurement errors, and noise. Our novel approach augments a loop-shaped inverse-Jacobian PI controller with a residual policy learned via Soft Actor-Critic in the digital twin. A delay-compensated Kalman filter fuses out-of-sequence camera measurements to provide latency-aware moving target position estimates. We compare the achieved performance to a performance bound for a class of model-based controllers derived from a frequency-domain sensitivity analysis, quantifying the minimum achievable error imposed by uncertain kinematic model mismatch. Simulation and hardware experiments indicate that our residual policy drives tracking error below the bound. Across conveyor speeds (27.9-41.8 mm/s), our method reduces mean tracking errors from 1.1-1.4 mm (baseline inverse-Jacobian PI) to 0.3-0.7 mm. These results show that learning residual corrections in a limited-fidelity simulator, combined with delay compensation, can substantially improve model-based control under modeling and measurement uncertainties while maintaining reliable sim-to-real transfer.
BibTeX
@misc{nobar2026residual,
  author = {Mahdi Nobar and Alisa Rupenyan and John Lygeros and Jürg Keller},
  title = {{Residual Reinforcement Learning for Visual Servo Control with Zero-Shot Sim-to-Real Transfer}},
  year = {2026},
  eprint = {2611.19318},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19318},
}
Figure