Flexible Gear Assembly with Visual Servoing and Force Feedback

Junjie Ming, Daniel Bargmann, Hongpeng Cao, Marco Caccamo

TL;DR

This paper presents a robust two-stage approach for high-precision (0.3 mm) robotic gear assembly. By combining YOLO for coarse localization with Deep Reinforcement Learning (DRL) for fine insertion—and bridging the gap with force feedback—the system completes complex assemblies in roughly 15 seconds, even when the target is partially obscured.

What Problem Does This Paper Address?

In industrial automation, gear assembly is a classic "high-precision" challenge. Traditional methods rely on "position teaching," where a human manually programs every coordinate. This is brittle; if the workpiece moves slightly, the process fails.

While vision-based systems (using cameras) and force-based systems (using touch) exist, they have a "blind spot." When a robot arm gets close enough to insert a gear, the on-wrist camera often loses a full view of the target (partial visibility). Furthermore, mapping complex force/torque signals to precise movements is mathematically difficult. The authors aim to create a flexible system that can "search" for a part and "insert" it accurately without tedious manual programming.

Methods

The authors propose a two-stage pipeline:

  1. Coarse Localization (The Search): The robot uses an on-wrist RGBD camera and the YOLO (You Only Look Once) algorithm to find the target platform. It uses depth data to calculate 3D coordinates and moves the robot to the general vicinity of the peg.
  2. Fine Alignment (The Insertion): Once the robot detects contact via a force sensor (force feedback), it switches to a DRL-based controller.
  3. DRL Algorithms: They compared DQN (Discrete actions) and PPO (Continuous actions).
  4. Partial Visibility Handling: Instead of requiring a full view of the peg, the DRL agent is trained to infer the relative position of the peg's center by looking at the edges and textures of the partially visible target platform.

Data Efficiency Strategy: To avoid the "data hunger" of deep learning, the authors used two clever tricks:

  • Synthetic Data: YOLO was trained entirely on synthetic images generated in Blender.
  • Offline Interaction Environment: Instead of training the DRL agent on a live robot for thousands of hours, they pre-recorded a grid of real-world images (a 30x35 mm "grid world"). The DRL agent "practiced" in this virtual environment built from real data before ever touching the physical robot.

Datasets

  • Synthetic RGB Images: Generated using 3D meshes of the gear platform in Blender for YOLO training.
  • Real-world Grid Map: A custom dataset consisting of images and coordinates sampled at 1mm intervals across a 3cm x 3.5cm area around the peg.
  • Evaluation Set: 100 randomly sampled starting positions in a real-world laboratory setup using a Franka Emika Panda robot.

Key Insights and Ideas

  • Continuous > Discrete: The PPO algorithm (continuous action space) significantly outperformed DQN. Because PPO can output smooth, precise movements, it was better at compensating for small calibration errors between the camera and the robot's gripper.
  • Force as a State Transitioner: Rather than using force to guide the entire movement, the authors used it as a robust "trigger" to tell the robot exactly when it had reached the platform and should switch from "searching" to "inserting."
  • Learning from the "In-Between": The most impressive "aha" moment is the DRL's ability to navigate based on partial views. By training on images where the peg is off-center or partially out of frame, the robot learns the "visual landscape" of the assembly site, allowing it to "know" where the center is even when it can't see it directly.

Key Results

  • Success Rate: The fine-tuned PPO model achieved a 100% success rate across 100 different starting positions.
  • Precision: The system successfully handled a 0.3 mm tolerance, which is standard for high-quality industrial gear fits.
  • Speed: The entire pipeline (searching + insertion) takes an average of 15 seconds. Specifically, the DRL-guided insertion phase takes only about 6 seconds.
  • Path Efficiency: The PPO agent achieved a "Traveling Score" (ATS) of 1.3, meaning its path was very close to a perfect straight line (a score of 1.0).

Limitations and Future Work

  • Visual Dependency: The method relies on "visually distinguishable spatial features." If the gear and the platform were perfectly uniform, featureless, and the same color, the DRL might struggle to orient itself.
  • 3D vs. 6D: The current work focuses on 3D space (X, Y, and Z-axis insertion). Real-world assembly often involves "cocking" or tilting (6 Degrees of Freedom). The authors plan to extend the pipeline to handle these complex rotations.
  • Grasping: In this study, the initial grasping of the gear from the tray was pre-programmed. Integrating a vision-based grasp into the start of the pipeline is a logical next step for full autonomy.

Summary updated

Abstract

Gear assembly is an essential but challenging task in industrial automation. This paper presents a novel two-stage approach for achieving high-precision and flexible gear assembly. The proposed approach integrates YOLO to coarsely localize the workpiece in a searching phase and deep reinforcement learning (DRL) to complete the insertion. Specifically, DRL addresses the challenge of partial visibility when the on-wrist camera is too close to the workpiece. Additionally, force feedback is used to smoothly transit the process from the first phase to the second phase. To reduce the data collection effort for training deep neural networks, we use synthetic RGB images for training YOLO and construct an offline interaction environment leveraging sampled real-world data for training DRL agents. We evaluate the proposed approach in a gear assembly experiment with a precision tolerance of 0.3mm. The results show that our method can robustly and efficiently complete searching and insertion from arbitrary positions within an average of 15 seconds.
BibTeX
@inproceedings{ming2023flexible,
  author = {Junjie Ming and Daniel Bargmann and Hongpeng Cao and Marco Caccamo},
  title = {{Flexible Gear Assembly with Visual Servoing and Force Feedback}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10341833},
}

Loading related content...

Figure