Segmentation of Objects by Hashing

J. de Curtò, I. de Zarzà, Alex Smola, Luc Van Gool

Abstract

We propose a novel approach to address the problem of Simultaneous Detection and Segmentation. Using hierarchical structures, we employ an efficient and accurate procedure that exploits feature information using Locality Sensitive Hashing (LSH). Building on convolutional neural networks for bounding box detection, we identify the top similar hierarchical region that best fits each box via hashing—an approach termed C&Z Segmentation. We further refine results through automatic hierarchical pruning. C&Z Segmentation offers a train-free alternative to Hypercolumns, achieving competitive state-of-the-art performance on the PASCAL VOC 2012 dataset.

Introduction

Simultaneous Detection and Segmentation (SDS) requires identifying both an object's location (bounding box) and its precise pixel-level mask. Traditional methods often rely on training deep neural networks on extensive hand-annotated datasets, which is computationally intensive. Models like "Hypercolumns" or complex proposal-based architectures frequently struggle to capture the rich, hierarchical part-based information inherent in objects. This research introduces a method to leverage image structure for mask generation without a dedicated supervised training phase for segmentation.

Methodology

The C&Z Segmentation pipeline consists of four primary stages:

  1. Detection: A Faster R-CNN with a VGG-16 backbone generates object bounding boxes.
  2. Hierarchy Generation: The image is decomposed into an Ultrametric Contour Map (UCM), creating a tree-like hierarchy of regions from superpixels to the full image.
  3. Hierarchical Section Hashing (HSH): Segmentation is treated as a retrieval task. Regions in the UCM hierarchy are indexed using Locality Sensitive Hashing (LSH). Detected bounding boxes are hashed to find the most similar hierarchical region in sublinear time.
  4. Hierarchical Section Pruning (HSP): Results are refined by pruning the hierarchy. If a retrieved mask is over-inclusive or poorly aligned, the system selects a more precise mid- or low-level region.

Experimental Evaluation

The approach was evaluated on the PASCAL VOC 2012 segmentation dataset. The VGG-16 detector was pre-trained on ImageNet and fine-tuned on VOC 2007 and 2012. Performance was measured against the VOC 2012 validation set using the Jaccard Index (Mean IoU).

Results and Discussion

C&Z Segmentation demonstrates that a search-based approach can compete with learning-based models. On the PASCAL VOC 2012 validation set, the model achieved a Mean IoU of 45.24% at the instance level and 43.05% at the class level.

Qualitative comparison between Ground Truth and C&Z Segmentation results across various PASCAL VOC categories.

The system effectively segments complex silhouettes, such as the intricate details of an eagle's head or the contours of a person, by identifying the optimal nodes in the image's hierarchical tree.

Detailed segmentation of a bald eagle using the C&Z approach.

Key Insights

  • Segmentation as Retrieval: Efficiently searching for pre-existing regions in an image hierarchy can bypass the need for pixel-level training.
  • Efficiency: As a "train-free" module, C&Z Segmentation can be integrated with any high-quality detector.
  • Computational Speed: LSH allows the system to overcome the dimensionality challenges of searching thousands of hierarchical regions.

Conclusion

While C&Z Segmentation is highly efficient, its performance is bounded by the accuracy of the initial bounding box detector and the quality of the UCM hierarchy. Future work may explore extending this hashing approach to 3D segmentation tasks, such as LIDAR data processing for autonomous systems, where real-time performance is paramount.

Summary updated

Abstract

We propose a novel approach to address the problem of Simultaneous Detection and Segmentation introduced in [Hariharan et al 2014]. Using the hierarchical structures first presented in [Arbeláez et al 2011] we use an efficient and accurate procedure that exploits the feature information of the hierarchy using Locality Sensitive Hashing. We build on recent work that utilizes convolutional neural networks to detect bounding boxes in an image [Ren et al 2015] and then use the top similar hierarchical region that best fits each bounding box after hashing, we call this approach C&Z Segmentation. We then refine our final segmentation results by automatic hierarchical pruning. C&Z Segmentation introduces a train-free alternative to Hypercolumns [Hariharan et al 2015]. We conduct extensive experiments on PASCAL VOC 2012 segmentation dataset, showing that C&Z gives competitive state-of-the-art segmentations of objects.
BibTeX
@misc{curto2017segmentation,
  author = {J. de Curtò and I. de Zarzà and Alex Smola and Luc Van Gool},
  title = {{Segmentation of Objects by Hashing}},
  year = {2017},
  eprint = {1702.08160},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1702.08160},
}
Figure