Abstract
Over the past several years there has been a considerable research investment into learning-based approaches to industrial assembly, but despite significant progress these techniques have yet to be adopted by industry. We argue that it is the prohibitively large design space for Deep Reinforcement Learning (DRL), rather than algorithmic limitations per se, that are truly responsible for this lack of adoption. Pushing these techniques into the industrial mainstream requires an industry-oriented paradigm which differs significantly from the academic mindset. In this paper we define criteria for industry-oriented DRL, and perform a thorough comparison according to these criteria of one family of learning approaches, DRL from demonstration, against a professional industrial integrator on the recently established NIST assembly benchmark. We explain the design choices, representing several years of investigation, which enabled our DRL system to consistently outperform the integrator baseline in terms of both speed and reliability. Finally, we conclude with a competition between our DRL system and a human on a challenge task of insertion into a randomly moving target. This study suggests that DRL is capable of outperforming not only established engineered approaches, but the human motor system as well, and that there remains significant room for improvement.
Introduction
In industrial manufacturing, robots traditionally rely on rigid, pre-programmed force or vision primitives to perform assembly tasks like connector insertion. While effective in static environments, these methods are expensive to set up and brittle to environmental perturbations. Deep Reinforcement Learning (DRL) promises to automate skill acquisition, yet it is rarely adopted in factories due to perceived complexity and unreliability. This paper introduces SHIELD (Super-Human InsErtion using Learning from Demonstration), a framework designed to prove that with the right design choices, DRL can be efficient, economical, and outperform both professional integrators and human operators.
The SHIELD Framework
SHIELD is built upon the Deep Deterministic Policy Gradient from Demonstration (DDPGfD) algorithm, with several key modifications tailored for industrial reliability.
The framework incorporates several strategic design choices:
- Simplified RL Mechanics: The policy is made deterministic by removing Gaussian exploration noise, and replay buffer prioritization is eliminated in favor of uniform sampling. This ensures the agent doesn't prematurely ignore crucial early demonstration data.
- On-Policy Human Correction: Similar to the DAGGER algorithm, human operators can intervene via a remote teaching device to help the robot escape local optima during training.
- Relative Coordinates: By treating the reset pose as the origin rather than using absolute robot base coordinates, the learned policy generalizes zero-shot to new locations on the board.
- Pre-trained Visual Features: A Variational Autoencoder (VAE) is pre-trained on teleoperation data to extract visual features, avoiding the time-intensive process of end-to-end training from pixels.
- Curriculum Learning: Training difficulty is scaled gradually, increasing starting pose offsets and target movement speeds over time.
Experimental Setup
The system was evaluated using the NIST Assembly Board Challenge, a standardized benchmark for industrial manipulation featuring various connectors such as USB, Waterproof, and DSUB.
Beyond the NIST board, the authors introduced custom challenge tasks, including a moving HDMI insertion task and a blind key-lock insertion task requiring force-based search. To support further research, a dataset of ~50 hours of robot interaction data and reward labels has been open-sourced.
Key Insights
The study reveals that simpler RL configurations can outperform complex ones in industrial settings. For instance, uniform sampling in the replay buffer prevents the agent from ignoring early demonstration data. Furthermore, human intervention serves as a critical "escape hatch" in sparse-reward environments. The use of relative coordinates is identified as a primary driver for generalization, allowing the robot to apply a learned "insertion template" across the workspace. Finally, the results demonstrate that RL can process multi-modal data (vision, force, proprioception) fast enough to match human reflexes in dynamic tasks.
Results
SHIELD demonstrated exceptional performance across several metrics:
- NIST Benchmark: Achieved a 99.8% success rate over 13,096 real-world trials.
- Industry Comparison: Outperformed two professional robotic integration vendors. While the vendors' solutions failed beyond a 2mm perturbation, the RL policy remained robust up to 5mm.
- Dynamic Insertion: In the moving HDMI task, the agent achieved a 100% success rate over 1,637 trials. Its mean completion time of 1,093ms was nearly identical to the human baseline of 1,041ms.
- Haptic Search: The agent successfully learned to perform "blind" insertions using only force and proprioception, maintaining high success rates (87-92%) despite significant starting offsets.
Conclusion and Future Work
While SHIELD shows great promise, limitations remain. The VAE-based vision model is sensitive to environmental changes like lighting or background distractions. Additionally, active hard contacts during insertion occasionally led to unpredictable dynamics within connectors. Future research will focus on developing gentler insertion policies, exploring offline RL to further reduce training time, and conducting larger-scale evaluations to achieve "five nines" (99.999%) reliability for industrial deployment.
Summary updated