Precise Pick-and-Place using Score-Based Diffusion Networks

Shih-Wei Guo, Tsu-Ching Hsiao, Yu-Lun Liu, Chun-Yi Lee

Introduction

Pick-and-place is a fundamental yet challenging task in industrial robotics, requiring high precision for successful assembly and manipulation. While traditional methods often rely on fiducial markers like AprilTags or precise 3D CAD models, these are not always practical in unstructured environments. Recent advancements in deep learning, such as Transporter Networks, have demonstrated the ability to learn from a few demonstrations. However, these models often discretize the action space—treating rotations as fixed bins—which inherently limits their precision. Furthermore, continuous pose estimation methods frequently necessitate expensive sensors like LiDAR or depth-based point clouds.

This paper introduces a novel framework that achieves continuous, high-precision pose estimation using only cost-effective top-down RGB images. By leveraging score-based diffusion models within a coarse-to-fine architecture, the system can achieve sub-millimeter and sub-degree accuracy even with extremely limited training data.

Methodology

Coarse-to-Fine Framework

The proposed approach utilizes a two-stage pipeline to handle the complexity of global search and local refinement:

  1. Coarse Stage: A global diffusion network processes the full workspace image to generate initial "rough" estimates for pick and place poses. This stage identifies the general location and orientation of the target objects.
  2. Fine Stage: To achieve high precision, the system crops "Oriented Regions of Interest" (ORoIs) around the coarse estimates. These crops are normalized by centering and rotating them to a canonical orientation. A second "fine" diffusion network then predicts the residual error, refining the pose to its exact coordinates.
The coarse-to-fine pipeline (a) and the internal architecture of the Pose Diffusion model (b).

Score-Based Diffusion on Lie Groups

Unlike standard regression models that may struggle with multi-modal distributions (e.g., symmetric objects), diffusion models learn to estimate the "score"—the gradient of the log-density of the data distribution. The model iteratively "denoises" a set of random initial poses, moving them along the learned score gradient until they converge on the most likely correct pose.

To ensure physical consistency, the authors adapt the diffusion process to operate on the Lie Group $SE(2)$, the natural mathematical space for 2D rigid body transformations. This geometric grounding allows the model to respect the underlying structure of rotations and translations in the physical world.

Visualization of the iterative denoising process (from $i=L$ to $i=1$), where initial noisy pose estimates converge toward the target object's true orientation.

Sample Efficiency and Augmentation

To address the challenge of learning from very few demonstrations (as few as one to ten), the framework employs aggressive pose and color augmentation. By artificially perturbing the positions, rotations, and appearances of objects in the training images, the model learns a robust representation of the task space without requiring an extensive real-world dataset.

Experimental Evaluation

The method was evaluated in both simulated and real-world environments:

  • Simulation: Using the Ravens simulator, the model was tested on an "L-shape-sim" task involving the insertion of an L-block into a matching frame.
  • Real-World: A UR5 robotic arm was used for six tasks, including an L-shape insertion and five LEGO DUPLO stacking tasks with varying geometries (e.g., fillets, eye-blocks).
  • Baselines: Performance was compared against state-of-the-art baselines like the Equivariant Transporter Network across different data scales (1, 10, and 100 demonstrations).

Results and Discussion

The experimental results highlight the framework's superior precision and sample efficiency:

  • High Success Rates: In the simulated L-shape task with 100 demonstrations, the method achieved a 93% success rate, matching or exceeding top baselines.
  • Low-Data Performance: With only 10 demonstrations, the proposed method significantly outperformed the Transporter Network in complex tasks like the orange-blue-fillet (95% vs. 21% success rate).
  • Rotational Precision: The fine-tuning stage proved essential for accuracy, reducing mean rotational errors to under 2 degrees in most tasks, whereas discrete baselines often suffered from significant errors due to symmetry confusion or binning artifacts.

Conclusion

This work demonstrates that high-precision robotic manipulation is achievable using cost-effective RGB sensors and minimal human demonstrations. By combining the global search capabilities of a coarse model with the local precision of a fine-grained diffusion refiner, the system overcomes the limitations of discrete action spaces. Future work includes extending the framework to $SE(3)$ to handle 3D poses and incorporating depth data to improve performance in height-variant environments.

Summary updated

BibTeX
@inproceedings{guo2024precise,
  author = {Shih-Wei Guo and Tsu-Ching Hsiao and Yu-Lun Liu and Chun-Yi Lee},
  title = {{Precise Pick-and-Place using Score-Based Diffusion Networks}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801772},
}
Figure