TL;DR
This paper introduces a symmetry-aware reinforcement learning (RL) framework for robotic peg-in-hole assembly using a compliant "soft wrist." By leveraging the geometric symmetry of parts through data augmentation and specialized auxiliary losses, the researchers enabled a memory-based agent to solve complex assembly tasks under partial observability, achieving real-world training in just three hours.
What Problem Does This Paper Address?
Robotic assembly, specifically the "peg-in-hole" task, is notoriously difficult due to the high precision required and the complex forces involved during contact. While rigid robots require high-frequency control to avoid damage, soft wrists offer a safer, more compliant alternative. However, soft components are non-linear and difficult to model.
Most existing RL solutions for this task assume "full observability"—meaning the robot knows exactly where the peg is relative to the hole via external cameras or motion capture. In real-world factories, occlusions and sensor limitations make this unrealistic. This paper tackles the POMDP (Partially Observable Markov Decision Process) setting, where the robot must rely solely on its own "sense of touch" (force/torque sensors) and internal joint data (proprioception). Because POMDPs are notoriously sample-inefficient to learn, the authors sought a way to bake "inductive biases" into the AI to speed up the process.
Methods
The authors propose a Symmetry-aware Recurrent Soft Actor-Critic (SAC) agent. The core idea is that if a square peg fits into a square hole in one orientation, it will also fit if you rotate both by 90 degrees or reflect them across an axis.
Key technical components include:
- Recurrent SAC: Uses a Recurrent Neural Network (RNN) to maintain a "memory" of past haptic interactions, allowing the agent to infer its relative position over time.
- Data Augmentation: During training, the agent takes successful or failed trajectories from its memory and mathematically transforms them (rotations and reflections) to create "new" experiences without needing more robot time.
- Symmetric Auxiliary Losses:
- Critic Loss: Forces the "Critic" (the part of the AI that predicts rewards) to give the same score to a trajectory and its symmetric counterpart.
- Actor Loss: Forces the "Actor" (the part that chooses actions) to choose a correspondingly rotated/reflected action when presented with a transformed history.
Datasets
The researchers evaluated their approach in both simulation and the real world:
- Simulation (MuJoCo): Five symmetric peg shapes (Triangle, Square, Pentagon, Hexagon, and Round) with a 1mm clearance.
- Real-World Hardware: A UR5e robot equipped with a custom soft wrist and a force/torque sensor.
- Demonstrations: They used 100 human-provided demonstrations (roughly 30 minutes of data) to "warm start" the learning process, helping the agent overcome the initial difficulty of finding the hole (reward sparsity).
Key Insights and Ideas
- Symmetry as a Shortcut: By enforcing symmetry, the agent doesn't have to "re-learn" how to insert a peg from every possible angle. This drastically reduces the search space for the optimal policy.
- Surpassing Humans: Interestingly, the learned policies became better than the human demonstrators. Humans tended to tilt the peg and "feel" for the hole (an MDP strategy), while the RL agent learned to use subtle vibrations and haptic feedback to slide directly to the center (a true POMDP strategy).
- Combining Techniques is Key: The ablation studies revealed that using either data augmentation or auxiliary losses helps, but using both provides a synergistic boost in performance and stability.
- Zero-Shot Generalization: A policy trained on a triangular peg could successfully insert a pentagonal or hexagonal peg without any additional training, proving the agent learned a general "insertion principle" rather than just memorizing a shape.
Key Results
- Real-World Speed: The agent learned the task directly on a real robot in under 3 hours (10,000 environment steps).
- Superior Performance: In simulations, the symmetry-aware agent (RSAC-Aug-Aux) consistently outperformed standard recurrent RL and even matched the performance of agents that had access to "cheating" (full-state) information.
- Robustness: The real-robot policy was robust enough to handle manual disruptions (someone pushing the arm) and could even transfer to a round peg with a tiny 50μm clearance (20x tighter than the training clearance).
Limitations and Future Work
The primary limitation is the requirement for near-perfect symmetry. The current math assumes the parts are geometrically regular. If a part is only "mostly" symmetric or completely irregular, the auxiliary losses might provide conflicting signals.
The authors suggest that future work could explore:
- Applying this to more complex, non-symmetric industrial parts.
- Integrating tactile (skin-like) sensors to provide even richer feedback than standard force/torque sensors.
- Relaxing the symmetry requirement to handle "imperfect" real-world objects.
Summary updated