Shape-Coded ArUco: Fiducial Marker for Bridging 2D and 3D Modalities

Lilika Makabe, Hiroaki Santo, Fumio Okura, Yasuyuki MatsushitaWACV 2022

TL;DR

This paper introduces a novel "shape-coded ArUco" marker that looks identical to standard 2D ArUco markers from above but has an embossed 3D shape. This allows for accurate 2D-to-3D registration, enabling precise color mapping onto untextured 3D scans, a task that is challenging with existing methods.

What Problem Does This Paper Address?

Accurately aligning 2D images with 3D models (2D-3D registration) is crucial for many applications, such as adding color and texture to 3D scans from laser scanners that don't capture this information. However, this is difficult because 2D images and 3D point clouds often lack common, easily detectable features. While fiducial markers (like ArUco) are great for 2D tasks, they don't inherently provide 3D information. Existing methods for untextured 3D data are either marker-less (less accurate) or rely on planar markers that require careful manual setup or have limitations in robustness. This paper aims to bridge this gap by creating a marker that works seamlessly with existing 2D ArUco detectors while also providing robust 3D information for precise registration.

Methods

The core innovation is the shape-coded ArUco marker. The authors achieve this by deforming a standard 2D ArUco marker. The key insight is that to maintain the same 2D appearance from any viewpoint, the deformation must satisfy specific conditions:

  • Boundary points (where colors meet) must remain on the original 2D plane (z=0).
  • Non-boundary points must be embossed downwards (z ≤ 0).

This embossing encodes depth information. The authors then developed a hybrid 2D-3D marker detection method:

  1. Point Cloud Clustering: The input 3D point cloud is clustered (using DBSCAN) to isolate potential marker regions.
  2. Height Segmentation: Each cluster is segmented into "top" and "embossed" parts based on their height along the normal direction, effectively creating a binary image representation.
  3. 2D Projection & Detection: The segmented point cloud is projected onto an image plane, and a standard 2D ArUco detector is used to identify the marker and its 2D corners.
  4. 3D Corner Refinement: The detected 2D corners are back-projected to the 3D point cloud. An initial 3D corner estimate is obtained, which is then refined using a Procrustes analysis with a synthetic marker model and an ICP-like refinement.

This process allows for the extraction of accurate 3D corner points from the shape-coded marker.

Datasets

The paper uses both synthetic and real-world datasets for evaluation.

  • Synthetic: A synthetic environment was created using Blender, rendering images of the shape-coded marker board from 50 viewpoints. Noise was added to the synthetic point clouds to test robustness.
  • Real-world: Real-world experiments involved capturing images of the shape-coded ArUco marker board with various untextured objects (a house, cake, car, and cube). 3D scans of these objects were acquired using a HandySCAN 3D laser scanner.

No new benchmarks were introduced; instead, the paper compares its method against existing approaches on these datasets.

Key Insights and Ideas

  • Projective Invariant Appearance: The core idea that a deformation can preserve the 2D appearance while encoding 3D information is crucial. This allows existing 2D detectors to work without modification.
  • Embossing for Depth: The specific choice of embossing downwards (z ≤ 0) for non-boundary points is key to avoiding occlusion and maintaining consistent color detection, a clever solution to a potential problem.
  • Hybrid 2D-3D Detection: Combining standard 2D ArUco detection with 3D point cloud processing (clustering, height segmentation, projection) provides a robust pipeline for finding markers in untextured 3D data.
  • Robustness to Noise: The shape-coded ArUco marker and its detection pipeline demonstrate remarkable robustness to noise in 3D point clouds, a common issue with laser scanners.

Key Results

  • Accurate 2D-3D Registration: The proposed method achieves significantly higher accuracy in camera pose estimation compared to a state-of-the-art 2D marker-based method, both in synthetic and real-world experiments.
  • Robustness: The method shows robust performance across varying levels of noise in 3D point clouds, outperforming the 2D marker method which degrades more rapidly.
  • Comparable to Multi-view: In real-world scenarios, the single-image results from the shape-coded ArUco are visually comparable to a multi-view 3D registration method that requires manual intervention.
  • 2D Detector Compatibility: The shape-coded ArUco marker yields comparable reprojection errors in 2D images to standard ArUco markers, confirming that the embossing does not negatively impact 2D detection.
  • Accurate 3D Corner Detection: The ICP-based refinement significantly improves 3D corner detection accuracy, achieving very low errors (e.g., median error of 0.002447 mm).

Limitations and Future Work

  • Noise Breakdown: The method starts to break down when the standard deviation of noise in 3D point clouds exceeds 1.3 mm. While this is still within the capabilities of many commercial scanners, it represents a limit.
  • Manual Intervention for Baselines: Some baseline methods (like the 2D marker method) required manual disambiguation or initial correspondences, which the proposed method avoids.
  • Future Work: The authors suggest that this shape-coding approach can be extended to other types of fiducial markers. The current work focuses on single-image input, and future work could explore multi-view extensions or more complex 3D deformations.

Summary updated

BibTeX
@inproceedings{makabe2022shape,
  author = {Lilika Makabe and Hiroaki Santo and Fumio Okura and Yasuyuki Matsushita},
  title = {{Shape-Coded ArUco: Fiducial Marker for Bridging 2D and 3D Modalities}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure