Abstract
Robotic assembly is widely utilized in large-scale manufacturing due to its high production efficiency, and the peg-in-hole assembly is a typical operation. While single peg-in-hole tasks have achieved great performance through reinforcement learning (RL) methods, multiple peg-in-hole assembly remains challenging due to complex geometry and physical constraints. To address this, we introduce a control policy workflow for multiple peg-in-hole assembly, dividing the task into three primitive sub-tasks: picking, alignment, and insertion to modularize the long-term task and improve sample efficiency. Sequential control policy (SeqPolicy), containing three control policies, is used to implement all the sub-tasks step-by-step. This approach introduces human knowledge to manage intermediate states, such as lifting height and aligning direction, thereby enabling flexible deployment across various scenarios. SeqPolicy demonstrated higher training efficiency with faster convergence and a higher success rate compared to the single control policy. Its adaptability is confirmed through generalization experiments involving objects with varying geometries. Recognizing the importance of object pose for control policies, a low-cost and adaptable method using visual representation containing objects’ pose information from RGB images is proposed to estimate objects’ pose in robot base frame directly in working scenarios. The representation is extracted by a Siamese-CNN network trained with self-supervised contrastive learning. Utilizing it, the alignment sub-task is successfully executed. These experiments validate the solution’s reusability and adaptability in multiple peg-in-hole scenarios.
Introduction
Robotic "peg-in-hole" assembly is a fundamental problem in industrial automation. While most research focuses on single-peg scenarios, real-world applications—such as assembling vehicle hubs, electronics, or medical equipment—often require the simultaneous alignment of multiple pegs. These tasks are exponentially more difficult because they involve higher degrees of freedom (up to 18 dimensions) and unforgiving physical constraints.
Traditional Reinforcement Learning (RL) often fails in these long-horizon tasks because the reward signal is sparse and the policy can easily get stuck in sub-optimal states, such as attempting to insert a peg before it is properly aligned. This paper introduces SeqPolicy, a modular framework designed to overcome these challenges through task decomposition and self-supervised vision.
Methodology
The core of the proposed solution is the Sequential Control Policy (SeqPolicy). Instead of training a single "end-to-end" network, the assembly process is divided into three distinct primitives: Picking, Alignment, and Insertion. Each sub-task is governed by its own RL policy trained using Proximal Policy Optimization (PPO).
Key Components:
- Modular RL: By breaking the task down, the system can incorporate human knowledge to define intermediate goals (e.g., specific lifting heights), which stabilizes the learning process.
- Self-Supervised Vision: For the alignment phase, a Siamese-CNN was developed. Trained via contrastive learning on pairs of RGB images, this network extracts low-dimensional visual representations that correlate with object poses without requiring manual labels.
- Compliance Control: To prevent hardware damage during contact, an impedance controller is integrated. This provides the necessary mechanical compliance to compensate for minor errors in the RL policy's output.
Experimental Setup and Results
The study utilized a simulation-to-reality pipeline using Isaac Gym, training 8,192 robots in parallel. To ensure robustness, the researchers employed domain randomization, varying lighting, mass, friction, and initial positions.
Performance Metrics
The modular approach demonstrated significant advantages over traditional methods:
- Efficiency: SeqPolicy achieved convergence in just 200 epochs, whereas single-policy models failed to reach the goal even after 1,000 epochs.
- Success Rate: The system achieved an overall success rate of over 80%. Specifically, it successfully assembled square (86.8%), triangular (78.7%), and elliptical (90.1%) pegs, proving its ability to generalize across different geometries.
- Compact Observations: Ablation studies revealed that a 13-dimensional observation space (including joint angles and hand force) was sufficient for high performance, drastically reducing computational overhead compared to high-dimensional raw inputs.
Conclusion
SeqPolicy offers a scalable and efficient solution for complex robotic assembly. By modularizing long-horizon tasks and utilizing self-supervised visual representations, the framework achieves high success rates and strong generalization capabilities. Future work will explore the integration of Large Language Models (LLMs) to manage policy transitions and the application of these techniques to unstructured real-world environments.
Summary updated