Canonical Plane Segmentation Without Annotating Pixel-Level Object Regions for Image Registration

Shunsuke Yoneda, Go Irie, Masashi NishiyamaIEEE Access 2024 · DOI

Abstract

In warehouse automation, precise object segmentation is crucial for robotic manipulation but typically requires labor-intensive pixel-level annotations. We present a weakly supervised framework that leverages 2D image registration to generate segmentation masks without manual labeling. By utilizing homography-based consistency and a structural similarity loss, our system achieves an 88% Intersection over Union (IoU) on warehouse datasets, significantly outperforming existing weakly supervised methods.

Introduction

Modern warehouse robots must recognize objects, estimate their 3D poses, and segment them from cluttered backgrounds to ensure successful picking. While deep learning has advanced recognition and pose estimation, high-quality segmentation remains a bottleneck due to the "annotation wall"—the need for humans to manually trace object boundaries for thousands of training images. Weakly supervised methods using only image-level labels often fail to capture precise object boundaries, typically focusing only on the most discriminative features (like logos) rather than the entire object.

Methodology: Registration as Supervision

The proposed framework bypasses manual annotation by turning image registration into a supervisory signal. The process involves four key stages:

  1. 2D Image Registration: Using feature matching (e.g., SuperPoint/SuperGlue), the system aligns a query image with a target image of the same object, calculating a homography matrix—a mathematical map that warps one image to align with the other.
  2. Pseudo-Region Generation: Instead of manual masks, the system initializes "pseudo-regions" by placing small circular masks around detected keypoints. This provides a rough, noisy starting point for the model.
  3. Segmentation Module: A deep encoder-decoder network takes the query image and these rough pseudo-regions as input to predict a refined mask.
  4. Consistency-Based Loss: The core innovation is a Region Loss based on Multi-Scale Structural Similarity (MSSIM). If the predicted mask is accurate, the pixels within the mask in the query image should match the warped pixels from the target image. Discrepancies caused by background pixels lower the similarity, forcing the network to refine the mask boundary to include only the object.
The top row shows planar objects in cluttered warehouse environments, while the bottom row displays the corresponding dominant canonical planes extracted by the proposed segmentation method.

Experimental Results

The system was evaluated on a composite dataset including YCB, APC, and ARC benchmarks across various background conditions (HOPE and HBDB).

  • Segmentation Performance: The method achieved an 88% IoU, a substantial improvement over other weakly supervised approaches like CONTA (75%) and PSA-Net (70%). While it does not yet reach the 99% accuracy of fully supervised models, it offers a high-performance alternative that requires zero manual masking.
  • Robustness: The framework maintained near-perfect recognition accuracy (~99%) while simultaneously performing pose estimation and segmentation.
  • Flexibility: The segmentation module is backend-agnostic, demonstrating compatibility with various registration tools including SIFT, LoFTR, and SuperPoint.

Conclusion and Future Work

By leveraging the inherent spatial constraints of image registration, this framework provides a scalable solution for robotic perception in dynamic environments. The "self-labeling" nature of the system allows for rapid deployment to new objects without the cost of human annotation.

One identified limitation is "Background Consistency"; if the background remains identical between the query and target images, the loss function may struggle to distinguish the object's edges. Future research will focus on boundary-specific loss terms and extending the method to complex 3D non-planar geometries.

Summary updated

BibTeX
@article{yoneda2024canonical,
  author = {Shunsuke Yoneda and Go Irie and Masashi Nishiyama},
  title = {{Canonical Plane Segmentation Without Annotating Pixel-Level Object Regions for Image Registration}},
  journal = {IEEE Access},
  year = {2024},
  doi = {10.1109/access.2024.3373463},
}
Figure