Symmetry-aware Reinforcement Learning for Robotic Assembly under Partial Observability with a Soft Wrist

Hai Nguyen, Tadashi Kozuno, Cristian C. Beltran-Hernandez, Masashi Hamaya

Abstract

This study addresses the contact-rich peg-in-hole task, a cornerstone of robotic assembly, by utilizing a compliant soft wrist. Unlike rigid systems, soft wrists offer increased safety and tolerance for lower-frequency control signals but introduce non-linear modeling complexities. We propose a partially observable Markov decision process (POMDP) formulation where a memory-based agent learns to perform assembly using only haptic and proprioceptive signals, eliminating the need for external pose estimators. To overcome the sample inefficiency typical of POMDPs, we introduce a symmetry-aware reinforcement learning (RL) framework. By leveraging geometric symmetries through data augmentation and auxiliary losses, our agent achieves high sample efficiency, enabling direct training on a real robot in under three hours. Results across various peg shapes demonstrate that our approach matches or exceeds the performance of state-based agents with full observability.

Introduction

Robotic assembly tasks, particularly the "peg-in-hole" problem, require extreme precision and the management of complex contact forces. Traditional approaches often rely on rigid robots and high-frequency control loops to prevent hardware damage. However, soft robotics offers a compelling alternative: a soft wrist provides passive compliance, allowing for safer interactions and more robust handling of uncertainties.

The challenge lies in the lack of full observability. In practical industrial settings, external cameras often face occlusions, making it impossible to know the exact relative pose of the peg and hole. Consequently, the robot must rely on its internal sensors—force/torque and proprioception—to "feel" its way through the task. This transforms the problem into a Partially Observable Markov Decision Process (POMDP), which is notoriously difficult to solve efficiently using standard RL techniques.

Methodology: Symmetry-aware Recurrent SAC

To address these challenges, we developed a Symmetry-aware Recurrent Soft Actor-Critic (SAC) agent. This framework incorporates two primary innovations: memory-based reasoning and geometric inductive biases.

Memory and Partial Observability

Since the agent cannot see the hole, it must infer the environment's state from a history of interactions. We utilize a Recurrent Neural Network (RNN) within the SAC architecture to maintain a temporal memory of haptic feedback. This allows the agent to distinguish between different contact states and build an internal representation of the peg's position relative to the hole over time.

Leveraging Geometric Symmetry

Industrial parts often possess inherent symmetries (e.g., square, hexagonal, or round pegs). We exploit this by enforcing symmetry in the learning process, effectively reducing the search space for the optimal policy. Our approach uses two mechanisms:

  1. Data Augmentation: During training, successful and failed trajectories are mathematically transformed (rotated or reflected) to create synthetic experiences. This allows the agent to learn from a single successful insertion as if it had occurred from multiple different approach angles.
  2. Symmetric Auxiliary Losses: We introduce specific constraints on the network's components:
  3. Critic Loss: Ensures the value function assigns identical scores to a trajectory and its symmetric counterparts.
  4. Actor Loss: Mandates that the policy chooses a correspondingly transformed action when presented with a transformed history.

Experimental Setup

We evaluated the framework in both a MuJoCo simulation and a real-world environment.

  • Simulation: We tested five symmetric shapes—Triangle, Square, Pentagon, Hexagon, and Round—with a tight 1mm clearance.
  • Real-World Hardware: A UR5e robotic arm was equipped with a custom soft wrist and a force/torque sensor.
  • Demonstrations: To mitigate the "sparse reward" problem (where the robot rarely finds the hole by chance), we provided the agent with 100 human demonstrations (approximately 30 minutes of data) to initialize the learning process.

Key Results and Insights

Efficiency and Performance

The inclusion of symmetry-aware components significantly boosted sample efficiency. The agent was able to learn the assembly task directly on the real robot in under three hours (approximately 10,000 environment steps). In simulations, our symmetry-aware agent (RSAC-Aug-Aux) consistently outperformed standard recurrent RL and reached performance levels comparable to agents that had access to "cheating" full-state information.

Emergent Strategies

Interestingly, the RL agent developed strategies that surpassed the human demonstrators. While humans typically used a "tilt-and-feel" strategy (an MDP-based approach), the RL agent learned to interpret subtle haptic vibrations and feedback to slide the peg directly to the center—a sophisticated POMDP strategy that minimizes contact time.

Robustness and Generalization

The trained policy demonstrated remarkable robustness, maintaining performance even when the robotic arm was manually disrupted during operation. Furthermore, we observed zero-shot generalization: a policy trained on a triangular peg could successfully insert a pentagonal or hexagonal peg without further training. In one extreme case, the real-robot policy successfully handled a round peg with a 50μm clearance—20 times tighter than the clearance used during its training phase.

Conclusion

By integrating symmetry as an inductive bias into a recurrent RL framework, we have demonstrated that complex, contact-rich assembly tasks can be solved under partial observability. The use of a soft wrist, combined with haptic-based learning, provides a path toward safer and more flexible industrial automation. Future work will focus on extending these principles to non-symmetric, irregular parts and integrating high-resolution tactile sensors to further enhance the robot's "sense of touch."

Summary updated

Abstract

This study tackles the representative yet challenging contact-rich peg-in-hole task of robotic assembly, using a soft wrist that can operate more safely and tolerate lower-frequency control signals than a rigid one. Previous studies often use a fully observable formulation, requiring external setups or estimators for the peg-to-hole pose. In contrast, we use a partially observable formulation and deep reinforcement learning from demonstrations to learn a memory-based agent that acts purely on haptic and proprioceptive signals. Moreover, previous works do not incorporate potential domain symmetry and thus must search for solutions in a bigger space. Instead, we propose to leverage the symmetry for sample efficiency by augmenting the training data and constructing auxiliary losses to force the agent to adhere to the symmetry. Results in simulation with five different symmetric peg shapes show that our proposed agent can be comparable to or even outperform a state-based agent. In particular, the sample efficiency also allows us to learn directly on the real robot within 3 hours.
BibTeX
@inproceedings{nguyen2024symmetry,
  author = {Hai Nguyen and Tadashi Kozuno and Cristian C. Beltran-Hernandez and Masashi Hamaya},
  title = {{Symmetry-aware Reinforcement Learning for Robotic Assembly under Partial Observability with a Soft Wrist}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=jiwzvVbpGi},
}
Figure