Abstract
Benchmarking provides experimental evidence of the scientific baseline to enhance the progression of fundamental research, which is also applicable to robotics. In this paper, we propose a method to benchmark metrics of robotic manipulation, which addresses the spatial-temporal reasoning skills for robot learning with the jigsaw game. In particular, our approach exploits a simple set of jigsaw pieces by designing a structured protocol, which can be highly customizable according to a wide range of task specifications. Researchers can selectively adopt the proposed protocol to benchmark their research outputs, on a comparable scale in the functional, task, and system-level of details. The purpose is to provide a potential look-up table for learning-based robot manipulation, commonly available in other engineering disciplines, to facilitate the adoption of robotics through calculated, empirical, and systematic experimental evidence.
Introduction
In the field of robot learning, reproducibility remains a significant challenge. Performance comparisons between different algorithms are often obscured by variations in hardware, sensor noise, and environmental conditions. While object sets like the Yale-CMU-Berkeley (YCB) collection have standardized the objects themselves, they do not standardize the task protocols or the reasoning required for complex manipulation. This paper introduces a standardized, low-cost benchmarking protocol based on jigsaw puzzles to provide a "look-up table" for robotic performance, requiring a combination of computer vision, precise motor control, and spatial-temporal logic.
Methodology
The "DeepClaw" Minimum Robot Cell
To minimize hardware-induced noise, we propose a simplified hardware template called DeepClaw. This setup consists of a manipulator (e.g., UR5, UR10e, or Franka Emika Panda), a simple suction cup end-effector, an overhead camera (e.g., Intel RealSense D435), and a flat work surface. This configuration serves as a baseline for evaluating manipulation software.
Structured Task Protocol
The manipulation process is decomposed into four functional modules, each evaluated with specific metrics:
- Segmentation: Partitioning the workspace image (Metric: Intersection over Union - IoU).
- Recognition: Identifying specific jigsaw pieces (Metric: Average Precision - AP).
- Pick Planning: Predicting optimal grasp poses (Metric: Success Rate).
- Motion Planning: Calculating collision-free paths (Metric: Grasping Time).
Tasks scale in complexity from simple Pick & Place to Tiling (2D arrangement) and Assembly (fitting pieces into a base plate).
Benchmarking Dataset
The Jigsaw Kit
The physical benchmarking kit consists of a low-cost 5-piece jigsaw set (one base plate and four fragmented pieces). The pieces can be customized with various textures, materials (wood, plastic, metal), and manufacturing methods to test visual and physical adaptability.
To ensure exact replication, we utilize a 6-dimensional descriptor (e.g., 000101) to normatively describe the specific jigsaw configuration used in any given trial.
Experimental Results and Insights
Hardware Transferability
Experiments across UR5, UR10e, and Panda platforms demonstrated that functional metrics (IoU and AP) remained consistent (within 5%) when using the same algorithm on similar hardware series, confirming the benchmark's transferability. However, the study also highlighted hardware sensitivity; for instance, the Panda robot achieved higher speeds but lower vision metrics due to its specific camera integration compared to the UR10e.
Spatial-Temporal Reasoning
Unlike standard bin-picking, the jigsaw task requires the robot to understand spatial relationships and temporal ordering. The "Assembly" task proved the most challenging, revealing that hardware errors (such as camera depth inaccuracy) often exceed assembly tolerances (<0.6mm). However, integrated vision-to-motion pipelines can compensate for these hardware limitations through dexterous manipulation strategies.
Conclusion and Future Work
This work establishes a baseline for 2D jigsaw assembly, proving that reliable benchmarking is possible even with low-precision hardware. Future research will address current limitations, such as the 2D constraint of flat pieces and the simplicity of suction-based picking. We aim to expand the protocol to include 6D grasping, multi-fingered grippers, and 3D jigsaw puzzles to further challenge robotic spatial reasoning and dexterity.
Summary updated