Robotic Sim-to-Real Transfer for Long-Horizon Pick-and-Place Tasks in the Robotic Sim2Real Competition

Ming Yang, Hongyu Cao, Lixuan Zhao, Chenrui Zhang, Yaran Chen

Abstract

Transferring robotic skills from simulation to the real world remains a significant hurdle due to discrepancies in sensing and actuation. In the ICRA 2024 Robotic Sim2Real (RSC) Challenge, the "ZeroBug" team demonstrated a robust engineering approach to bridge this gap. By implementing a motion-blur-resistant visual pipeline and a feedback-linearized servo system, the team achieved a 100% success rate in real-world long-horizon pick-and-place tasks, securing first place in the competition. This paper details their modular strategy, which prioritizes classical control and targeted perception over complex reinforcement learning.

1. Introduction

The "sim-to-real gap" is a persistent challenge in robotics, where policies trained in high-fidelity simulators often fail when deployed on physical hardware. These failures typically stem from two primary sources: sensing discrepancies, such as motion blur and lighting variations, and actuation discrepancies, including motor dead zones and friction. While Reinforcement Learning (RL) with domain randomization is a popular research direction, it often requires extensive data and can be unpredictable. This work presents an alternative: a modular, principled engineering framework that achieves high-performance transfer without environment-specific code modifications.

2. Addressing Sensing Discrepancies: SMMS

Mobile robots frequently encounter motion blur, which significantly degrades the performance of standard visual markers like ArUco. To counter this, the authors developed the Sequential Motion-blur Mitigation Strategy (SMMS).

2.1 Image Enhancement and Classification

The perception pipeline begins with targeted image processing, boosting the red channel and subtracting the green channel to isolate the "mineral" markers used in the competition. For classification, the team replaced traditional template matching with a lightweight CNN (modeled after LeNet-5). This CNN achieved inference speeds of approximately 11.14 ms, allowing for real-time reactive control.

2.2 Robustness and Filtering

To ensure reliability, the system incorporates a recognition rejection threshold to filter false positives. Furthermore, a low-pass filter combined with extrapolation techniques allows the robot to maintain a "lock" on targets even during frames where the marker is severely blurred or temporarily lost. This approach increased real-world recall from 50% to 70%.

3. Robust Control via Feedback Linearization

Actuation discrepancies, particularly motor non-linearities and dead zones, often lead to steady-state errors in real-world deployment. The authors addressed this using a feedback-linearized servo system for the robot's omnidirectional chassis.

3.1 Design Functions (DF)

The team evaluated three distinct Design Functions to determine the optimal error attenuation strategy:

  • Type I (Linear): A standard linear approach, which proved sensitive to motor dead zones.
  • Type II (Power-law): Designed to boost output at small error margins, yet still lacked sufficient precision.
  • Type III (Integral): This function incorporated an integral term to eliminate steady-state error.

The Type III controller emerged as the most robust, achieving sub-centimeter accuracy (0.81 cm in x, 0.63 cm in y) in the physical arena.

4. Experimental Results

The system was validated within the 2024 Robotic Sim2Real Challenge framework, utilizing the Habitat simulator for training and a physical EP robot for real-world testing.

4.1 Performance Metrics

The perception pipeline's efficiency (11.14 ms/frame) enabled high-speed operation. In the final competition phase, the robot demonstrated exceptional reliability, achieving a 100% success rate (6/6) for both grasping and three-layer stacking in the real world.

4.2 The "Penetration" Paradox

An interesting finding during testing was the "penetration" paradox. Simulators often allow grippers to slightly penetrate object meshes, leading to successful grasps in sim that would cause physical collisions and failures in reality. The high-precision Type III servo system mitigated this by ensuring near-perfect alignment, preventing "diagonal" grasp failures.

5. Conclusion and Future Work

The ZeroBug team's success at ICRA 2024 underscores the efficacy of modular engineering in solving sim-to-real challenges. By specifically targeting motion blur and actuation non-linearities, they achieved superior performance compared to many black-box learning methods. Future work will focus on generalizing the perception pipeline beyond color-specific markers and adapting the system for high-speed, unstructured industrial environments.

Summary updated

Abstract

This paper presents a fully autonomous robotic system that performs sim-to-real transfer in complex longhorizon tasks involving navigation, recognition, grasping, and stacking in an environment with multiple obstacles. The key feature of the system is the ability to overcome typical sensing and actuation discrepancies during sim-to-real transfer and to achieve consistent performance without any algorithmic modifications. To accomplish this, a lightweight noise-resistant visual perception system and a nonlinearityrobust servo system are adopted. We conduct a series of tests in both simulated and realworld environments. The visual perception system achieves the speed of 11 ms per frame due to its lightweight nature, and the servo system achieves sub-centimeter accuracy with the proposed controller. Both exhibit high consistency during sim-to-real transfer. Benefiting from these, our robotic system took first place in the mineral searching task of the Robotic Sim2Real Challenge hosted at ICRA 2024.
BibTeX
@inproceedings{yang2025robotic,
  author = {Ming Yang and Hongyu Cao and Lixuan Zhao and Chenrui Zhang and Yaran Chen},
  title = {{Robotic Sim-to-Real Transfer for Long-Horizon Pick-and-Place Tasks in the Robotic Sim2Real Competition}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2025},
  url = {https://openreview.net/forum?id=jbrCaLi9CX},
}
Figure