Symmetry-aware Reinforcement Learning for Robotic Assembly under Partial Observability with a Soft Wrist

Hai Nguyen, Tadashi Kozuno, Cristian C. Beltran-Hernandez, Masashi Hamaya

TL;DR

This paper introduces a symmetry-aware reinforcement learning (RL) framework for robotic peg-in-hole assembly using a compliant "soft wrist." By leveraging the geometric symmetry of parts through data augmentation and specialized auxiliary losses, the researchers enabled a memory-based agent to solve complex assembly tasks under partial observability, achieving real-world training in just three hours.

What Problem Does This Paper Address?

Robotic assembly, specifically the "peg-in-hole" task, is notoriously difficult due to the high precision required and the complex forces involved during contact. While rigid robots require high-frequency control to avoid damage, soft wrists offer a safer, more compliant alternative. However, soft components are non-linear and difficult to model.

Most existing RL solutions for this task assume "full observability"—meaning the robot knows exactly where the peg is relative to the hole via external cameras or motion capture. In real-world factories, occlusions and sensor limitations make this unrealistic. This paper tackles the POMDP (Partially Observable Markov Decision Process) setting, where the robot must rely solely on its own "sense of touch" (force/torque sensors) and internal joint data (proprioception). Because POMDPs are notoriously sample-inefficient to learn, the authors sought a way to bake "inductive biases" into the AI to speed up the process.

Methods

The authors propose a Symmetry-aware Recurrent Soft Actor-Critic (SAC) agent. The core idea is that if a square peg fits into a square hole in one orientation, it will also fit if you rotate both by 90 degrees or reflect them across an axis.

Key technical components include:

  • Recurrent SAC: Uses a Recurrent Neural Network (RNN) to maintain a "memory" of past haptic interactions, allowing the agent to infer its relative position over time.
  • Data Augmentation: During training, the agent takes successful or failed trajectories from its memory and mathematically transforms them (rotations and reflections) to create "new" experiences without needing more robot time.
  • Symmetric Auxiliary Losses:
    • Critic Loss: Forces the "Critic" (the part of the AI that predicts rewards) to give the same score to a trajectory and its symmetric counterpart.
  • Actor Loss: Forces the "Actor" (the part that chooses actions) to choose a correspondingly rotated/reflected action when presented with a transformed history.

Datasets

The researchers evaluated their approach in both simulation and the real world:

  • Simulation (MuJoCo): Five symmetric peg shapes (Triangle, Square, Pentagon, Hexagon, and Round) with a 1mm clearance.
  • Real-World Hardware: A UR5e robot equipped with a custom soft wrist and a force/torque sensor.
  • Demonstrations: They used 100 human-provided demonstrations (roughly 30 minutes of data) to "warm start" the learning process, helping the agent overcome the initial difficulty of finding the hole (reward sparsity).

Key Insights and Ideas

  • Symmetry as a Shortcut: By enforcing symmetry, the agent doesn't have to "re-learn" how to insert a peg from every possible angle. This drastically reduces the search space for the optimal policy.
  • Surpassing Humans: Interestingly, the learned policies became better than the human demonstrators. Humans tended to tilt the peg and "feel" for the hole (an MDP strategy), while the RL agent learned to use subtle vibrations and haptic feedback to slide directly to the center (a true POMDP strategy).
  • Combining Techniques is Key: The ablation studies revealed that using either data augmentation or auxiliary losses helps, but using both provides a synergistic boost in performance and stability.
  • Zero-Shot Generalization: A policy trained on a triangular peg could successfully insert a pentagonal or hexagonal peg without any additional training, proving the agent learned a general "insertion principle" rather than just memorizing a shape.

Key Results

  • Real-World Speed: The agent learned the task directly on a real robot in under 3 hours (10,000 environment steps).
  • Superior Performance: In simulations, the symmetry-aware agent (RSAC-Aug-Aux) consistently outperformed standard recurrent RL and even matched the performance of agents that had access to "cheating" (full-state) information.
  • Robustness: The real-robot policy was robust enough to handle manual disruptions (someone pushing the arm) and could even transfer to a round peg with a tiny 50μm clearance (20x tighter than the training clearance).

Limitations and Future Work

The primary limitation is the requirement for near-perfect symmetry. The current math assumes the parts are geometrically regular. If a part is only "mostly" symmetric or completely irregular, the auxiliary losses might provide conflicting signals.

The authors suggest that future work could explore:

  1. Applying this to more complex, non-symmetric industrial parts.
  2. Integrating tactile (skin-like) sensors to provide even richer feedback than standard force/torque sensors.
  3. Relaxing the symmetry requirement to handle "imperfect" real-world objects.

Summary updated

Abstract

This study tackles the representative yet challenging contact-rich peg-in-hole task of robotic assembly, using a soft wrist that can operate more safely and tolerate lower-frequency control signals than a rigid one. Previous studies often use a fully observable formulation, requiring external setups or estimators for the peg-to-hole pose. In contrast, we use a partially observable formulation and deep reinforcement learning from demonstrations to learn a memory-based agent that acts purely on haptic and proprioceptive signals. Moreover, previous works do not incorporate potential domain symmetry and thus must search for solutions in a bigger space. Instead, we propose to leverage the symmetry for sample efficiency by augmenting the training data and constructing auxiliary losses to force the agent to adhere to the symmetry. Results in simulation with five different symmetric peg shapes show that our proposed agent can be comparable to or even outperform a state-based agent. In particular, the sample efficiency also allows us to learn directly on the real robot within 3 hours.
BibTeX
@inproceedings{nguyen2024symmetry,
  author = {Hai Nguyen and Tadashi Kozuno and Cristian C. Beltran-Hernandez and Masashi Hamaya},
  title = {{Symmetry-aware Reinforcement Learning for Robotic Assembly under Partial Observability with a Soft Wrist}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=jiwzvVbpGi},
}

Loading related content...

Figure