Robotic Object Insertion with a Soft Wrist through Sim-to-Real Privileged Training

Yuni Fuchioka, Cristian C. Beltran-Hernandez, Hai Nguyen, Masashi Hamaya

TL;DR

Researchers have developed a robust sim-to-real framework for contact-rich "peg-in-hole" tasks using a robot equipped with a passive soft wrist. By employing a two-stage "privileged training" strategy—where a student agent learns to estimate hidden physical states from sensor history—the system achieves high success rates in the real world without requiring any real-world training data or external motion-capture systems.

What Problem Does This Paper Address?

Precision assembly tasks, like inserting a peg into a hole, are notoriously difficult for robots when there is uncertainty in the object's grasp or the hole's exact location. While "soft" or compliant wrists are excellent for these tasks because they allow for safe, passive deformation during contact, they introduce a major control challenge: the wrist's deformation is nonlinear and cannot be directly measured by standard robot sensors.

Traditionally, researchers solve this by using expensive external motion-capture systems or by collecting massive amounts of interaction data in the real world, which is time-consuming and can damage hardware. This paper tackles the challenge of achieving high-precision insertion using a soft wrist while relying only on simulation for training and standard onboard sensors (arm pose and force/torque data) for execution.

Methods

The authors utilize a Sim-to-Real Privileged Training approach, which breaks the learning process into two distinct phases:

  1. Teacher Training (RL): A "teacher" policy is trained in a MuJoCo-based simulation using Reinforcement Learning (Proximal Policy Optimization, or PPO). Because it's a simulation, the teacher has access to "privileged information" that a real robot wouldn't have, such as the exact relative pose of the peg and the hole, and a binary "alignment signal."
  2. Student Training (Imitation): A "student" agent is then trained to mimic the teacher. However, the student does not have access to the privileged information. Instead, it uses a Temporal Convolutional Network (TCN) to analyze a one-second history of standard sensor data (wrist position and force/torque readings) to estimate the privileged states.

To ensure the policy works on a physical robot, the authors used extensive Domain Randomization. They randomized the peg's in-grasp angle, the hole's position, and—crucially—the gains of the Operational Space Control (OSC) to account for the mechanical differences between a simulated torque-controlled robot and a stiff, position-controlled real-world UR5e.

Datasets

As an RL-based study, the "dataset" consists of the simulation environment and the physical evaluation setup:

  • Simulation: Built on Robosuite and MuJoCo, featuring a custom-modeled soft wrist consisting of four springs.
  • Physical Evaluation: Conducted on a Universal Robots UR5e with a Robotiq Hand-E gripper and a custom 3D-printed soft wrist.
  • Objects: Training was performed exclusively on a circular peg (40mm diameter). Testing included the circular peg and an unseen square peg to evaluate generalization.

Key Insights and Ideas

  • The "Alignment Signal" is Critical: Including a binary signal in the privileged information that tells the teacher if the peg is vertically aligned with the hole proved vital. It acts as a subtask switching condition, helping the policy learn when to transition from "searching" for the hole to "inserting" into it.
  • Estimating Physics vs. Latent Spaces: Unlike some prior works that use abstract "latent embeddings," this student encoder directly regresses onto physical quantities (like peg pose). This makes the system more interpretable and easier to debug.
  • Soft Wrists as Low-Pass Filters: The authors observed that the soft wrist's mechanical structure naturally filtered out high-frequency torque noise, allowing them to ignore torque readings and focus solely on 3D force data for the control policy.
  • Zero-Shot Generalization: The policy learned on a circular peg generalized surprisingly well to a square peg, suggesting the RL agent learned fundamental contact-handling strategies rather than just memorizing a specific geometry.

Key Results

The system demonstrated impressive zero-shot transfer from simulation to the real world:

  • Circular Peg: Achieved a 100% success rate with 0° misalignment and an 80-95% success rate with ±5° misalignment, even when the starting position was shifted by ±10mm.
  • Square Peg (Unseen): Despite never seeing a square peg during training, the "best" network achieved a 70% success rate in the 0° misalignment condition.
  • Simulation Advantage: In head-to-head sim tests, the privileged training approach significantly outperformed a baseline TCN that tried to learn the task directly from sensor data without the teacher-student decomposition.

Limitations and Future Work

The primary failure mode observed was the peg "missing" the hole entirely. Because the policy only attempts insertion after detecting a specific contact impact, if the initial search fails to find the hole's edge, the robot may never trigger the insertion phase.

The authors suggest that future work could involve:

  • Randomizing more physical parameters like surface friction and spring damping to further close the sim-to-real gap.
  • Adding reward penalties to encourage faster task completion.
  • Testing the system on more complex, tighter-tolerance industrial components like electrical connectors.

Summary updated

Abstract

This study addresses contact-rich object insertion tasks under unstructured environments using a robot with a soft wrist, enabling safe contact interactions. For the unstructured environments, we assume that there are uncertainties in object grasp and hole pose and that the soft wrist pose cannot be directly measured. Recent methods employ learning approaches and force/torque sensors for contact localization; however, they require data collection in the real world. This study proposes a sim-to-real approach using a privileged training strategy. This method has two steps. 1) The teacher policy is trained to complete the task with sensor inputs and ground truth privileged information such as the peg pose, and then 2) the student encoder is trained with data produced from teacher policy rollouts to estimate the privileged information from sensor history. We performed sim-to-real experiments under grasp and hole pose uncertainties. This resulted in 100%, 95%, and 80% success rates for circular peg insertion with 0°, +5°, and -5° peg misalignments, respectively, and start positions randomly shifted ± 10 mm from a default position. Also, we tested the proposed method with a square peg that was never seen during training. Additional simulation evaluations revealed that using the privileged strategy improved success rates compared to training with only simulated sensor data. Our results demonstrate the advantage of using sim-to-real privileged training for soft robots, which has the potential to alleviate human engineering efforts for robotic assembly.
BibTeX
@inproceedings{fuchioka2024robotic,
  author = {Yuni Fuchioka and Cristian C. Beltran-Hernandez and Hai Nguyen and Masashi Hamaya},
  title = {{Robotic Object Insertion with a Soft Wrist through Sim-to-Real Privileged Training}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=e9HKbLMjY4},
}

Loading related content...

Figure