Embedded Sim-to-Real Visual Servoing: Learned Reaching and Force-Guided Placement

Dezhi Sun, Duy Khang Banh, Kenan NiuarXiv 2026 · arXiv

This paper presents an embedded architecture for robotic manipulation that bridges the "reality gap" by combining learned sim-to-real reaching with force-guided analytical placement. By utilizing a compact state-based interface rather than raw pixels, the authors achieve reliable real-world performance on an NVIDIA Jetson platform, maintaining high perception throughput for dynamic control.

TL;DR

The researchers developed a system that splits robotic manipulation into two regimes: learned free-space reaching and force-guided contact correction. By feeding metric, reconstructed targets into compact neural network policies, they achieved a 70% success rate in real-world "sorting" tasks without any real-world fine-tuning, while maintaining a perception latency of just 34.5 ms.

What Problem Does This Paper Address?

Robotic manipulation in the real world faces a triple threat: tight computational constraints on embedded hardware, the "reality gap" when transferring policies from simulation, and the inherent imprecision of vision for contact-rich tasks (like placing a block in a tight hole).

While large-scale visuomotor policies are popular, they often require massive compute and struggle with the subtle physics of contact. Conversely, purely analytical methods can be brittle in unstructured environments. This paper explores how to combine the best of both worlds—learned "reaching" and analytical "placement"—on a shared embedded platform (NVIDIA Jetson AGX Orin) controlling a Franka Research 3 (FR3) robot.

Methods

The core of the approach is a shared embedded front end that reconstructs metric targets from eye-in-hand RGB-D observations.

  1. Perception: A YOLO11-seg model runs on the Jetson Orin via TensorRT to segment objects. The system then back-projects mask centroids into 3D space using depth data and hand-eye calibration to find the target's position in the robot's base frame.
  2. The "Bridge" (State-based Interface): Instead of feeding raw images to the reinforcement learning (RL) policy, the authors use a 15-dimensional vector containing joint positions, end-effector pose, object identity, and the reconstructed 3D target. This "externalizes appearance," meaning the RL policy only deals with metric coordinates, significantly reducing domain shift.
  3. Learned Hybrid Strategy: Uses two Proximal Policy Optimization (PPO) policies trained in the RLBench simulator. Policy A handles object reaching, while Policy B handles coarse hole-region approach. Scripted logic manages the grasping and lifting between these stages.
  4. Force-Guided Analytical Strategy: A baseline that uses deterministic Cartesian transport followed by a spiral search once contact is detected (~2 N normal force). This allows the robot to "feel" its way into the hole, correcting for visual localization errors.
Motion-stage decomposition showing the learned hybrid and force-guided analytical pipelines.

Datasets

  • Simulation: Policies were trained and evaluated using RLBench (built on PyRep and CoppeliaSim). Each policy was tested over 1,000 simulation episodes.
  • Perception Training: The YOLO11-seg model was trained on a custom dataset of approximately 100 labeled RGB-D images of the workspace, featuring four classes: Blue circle, Green triangle, Nijntje/Miffy, and Yellow square.

Key Insights and Ideas

  • Decoupling Perception from Policy: By converting pixels to metric coordinates before they hit the neural network, the authors bypassed the need for complex domain randomization of textures and lighting. The policy simply sees "where the object is," not "what it looks like."
  • The Post-processing Bottleneck: Interestingly, the researchers found that post-processing (mask decoding and centroid extraction) took longer (15.3 ms) than the actual neural network inference (11.1 ms). This highlights that for embedded robotics, optimizing the "glue" code is just as important as optimizing the model.
  • Regime-Aware Architecture: The results suggest that while RL is excellent for the "macro" movement of reaching through free space, explicit force-feedback loops are still superior for the "micro" movements of precise placement.

Key Results

The system was evaluated across 100 physical trials (50 per strategy).

  • Success Rates: The learned hybrid pipeline achieved a 70.0% completion rate, while the force-guided analytical pipeline achieved 74.0%. The overlapping confidence intervals suggest comparable performance in the evaluated setup.
  • Simulation Performance: In the simulator, the PPO policies achieved 87.2% (Policy A) and 83.1% (Policy B) success rates.
  • Latency: The total mean perception latency was 34.55 ms, enabling a throughput of approximately 28.9 FPS, which is critical for reactive visual servoing.
Quantitative evaluation of simulation outcomes, real-robot outcomes, and perception timing.

Limitations and Future Work

The authors identified three main "vulnerabilities":

  1. Self-Occlusion: The wrist-mounted camera can lose sight of the object as the gripper gets very close, leading to stale target estimates.
  2. Perception Jumps: False positives in the segmentation model can cause the target to "jump," leading to discontinuous robot commands.
  3. Orientation Sensitivity: The current study focused on approximately aligned objects. Future work is needed to handle arbitrary 3D orientations and more complex geometries.

The authors conclude that these results motivate a "regime-aware" architecture that explicitly transitions from learned free-space motion to force-based correction upon contact.

Summary updated

Abstract

Embedded perception-to-control systems must combine visual targeting with reliable contact handling under tight computational constraints. We present a shared architecture on an NVIDIA Jetson AGX Orin controlling a Franka Research 3 through Cartesian impedance control. RealSense D435i observations, YOLO11-seg segmentation, depth back-projection, and hand--eye calibration provide metric robot-frame targets. We compare two strategies: compact proximal policy optimization (PPO) policies transferred from simulation for object- and hole-region reaching, and force-guided Cartesian placement using approximately 2 N normal contact and spiral search. The PPO policies achieved 87.2% and 83.1% success over 1000 simulation episodes each. Across 50 real-robot trials per strategy, the learned hybrid and force-guided analytical pipelines achieved 70.0% and 74.0% completion, respectively. Mean perception latency with TensorRT was 34.55 ms. These results support an embedded architecture combining learned free-space reaching with explicit contact correction.
BibTeX
@misc{sun2026embedded,
  author = {Dezhi Sun and Duy Khang Banh and Kenan Niu},
  title = {{Embedded Sim-to-Real Visual Servoing: Learned Reaching and Force-Guided Placement}},
  year = {2026},
  eprint = {2611.15658},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15658},
}
Figure