Abstract
This study addresses the contact-rich peg-in-hole task, a cornerstone of robotic assembly, by utilizing a compliant soft wrist. Unlike rigid systems, soft wrists offer increased safety and tolerance for lower-frequency control signals but introduce non-linear modeling complexities. We propose a partially observable Markov decision process (POMDP) formulation where a memory-based agent learns to perform assembly using only haptic and proprioceptive signals, eliminating the need for external pose estimators. To overcome the sample inefficiency typical of POMDPs, we introduce a symmetry-aware reinforcement learning (RL) framework. By leveraging geometric symmetries through data augmentation and auxiliary losses, our agent achieves high sample efficiency, enabling direct training on a real robot in under three hours. Results across various peg shapes demonstrate that our approach matches or exceeds the performance of state-based agents with full observability.
Introduction
Robotic assembly tasks, particularly the "peg-in-hole" problem, require extreme precision and the management of complex contact forces. Traditional approaches often rely on rigid robots and high-frequency control loops to prevent hardware damage. However, soft robotics offers a compelling alternative: a soft wrist provides passive compliance, allowing for safer interactions and more robust handling of uncertainties.
The challenge lies in the lack of full observability. In practical industrial settings, external cameras often face occlusions, making it impossible to know the exact relative pose of the peg and hole. Consequently, the robot must rely on its internal sensors—force/torque and proprioception—to "feel" its way through the task. This transforms the problem into a Partially Observable Markov Decision Process (POMDP), which is notoriously difficult to solve efficiently using standard RL techniques.
Methodology: Symmetry-aware Recurrent SAC
To address these challenges, we developed a Symmetry-aware Recurrent Soft Actor-Critic (SAC) agent. This framework incorporates two primary innovations: memory-based reasoning and geometric inductive biases.
Memory and Partial Observability
Since the agent cannot see the hole, it must infer the environment's state from a history of interactions. We utilize a Recurrent Neural Network (RNN) within the SAC architecture to maintain a temporal memory of haptic feedback. This allows the agent to distinguish between different contact states and build an internal representation of the peg's position relative to the hole over time.
Leveraging Geometric Symmetry
Industrial parts often possess inherent symmetries (e.g., square, hexagonal, or round pegs). We exploit this by enforcing symmetry in the learning process, effectively reducing the search space for the optimal policy. Our approach uses two mechanisms:
- Data Augmentation: During training, successful and failed trajectories are mathematically transformed (rotated or reflected) to create synthetic experiences. This allows the agent to learn from a single successful insertion as if it had occurred from multiple different approach angles.
- Symmetric Auxiliary Losses: We introduce specific constraints on the network's components:
- Critic Loss: Ensures the value function assigns identical scores to a trajectory and its symmetric counterparts.
- Actor Loss: Mandates that the policy chooses a correspondingly transformed action when presented with a transformed history.
Experimental Setup
We evaluated the framework in both a MuJoCo simulation and a real-world environment.
- Simulation: We tested five symmetric shapes—Triangle, Square, Pentagon, Hexagon, and Round—with a tight 1mm clearance.
- Real-World Hardware: A UR5e robotic arm was equipped with a custom soft wrist and a force/torque sensor.
- Demonstrations: To mitigate the "sparse reward" problem (where the robot rarely finds the hole by chance), we provided the agent with 100 human demonstrations (approximately 30 minutes of data) to initialize the learning process.
Key Results and Insights
Efficiency and Performance
The inclusion of symmetry-aware components significantly boosted sample efficiency. The agent was able to learn the assembly task directly on the real robot in under three hours (approximately 10,000 environment steps). In simulations, our symmetry-aware agent (RSAC-Aug-Aux) consistently outperformed standard recurrent RL and reached performance levels comparable to agents that had access to "cheating" full-state information.
Emergent Strategies
Interestingly, the RL agent developed strategies that surpassed the human demonstrators. While humans typically used a "tilt-and-feel" strategy (an MDP-based approach), the RL agent learned to interpret subtle haptic vibrations and feedback to slide the peg directly to the center—a sophisticated POMDP strategy that minimizes contact time.
Robustness and Generalization
The trained policy demonstrated remarkable robustness, maintaining performance even when the robotic arm was manually disrupted during operation. Furthermore, we observed zero-shot generalization: a policy trained on a triangular peg could successfully insert a pentagonal or hexagonal peg without further training. In one extreme case, the real-robot policy successfully handled a round peg with a 50μm clearance—20 times tighter than the clearance used during its training phase.
Conclusion
By integrating symmetry as an inductive bias into a recurrent RL framework, we have demonstrated that complex, contact-rich assembly tasks can be solved under partial observability. The use of a soft wrist, combined with haptic-based learning, provides a path toward safer and more flexible industrial automation. Future work will focus on extending these principles to non-symmetric, irregular parts and integrating high-resolution tactile sensors to further enhance the robot's "sense of touch."
Summary updated