HashBox: Hash Hierarchical Segmentation exploiting Bounding Box Object Detection

Joachim de Curtò, Irene Zarza, Alexander J. Smola, Luc Van Gool

Introduction

Simultaneous Detection and Segmentation (SDS) represents a core challenge in computer vision, requiring both the localization of objects via bounding boxes and the precise identification of their constituent pixels through masks. Traditional approaches, such as Mask R-CNN, typically rely on training specialized neural network branches to predict these masks. However, this paradigm necessitates vast amounts of pixel-level annotations and significant computational resources.

The "C&Z Segmentation" framework, also known as HashBox, proposes a paradigm shift by treating segmentation as a retrieval problem rather than a generative one. By leveraging existing object detectors and efficient hashing techniques over hierarchical image structures, HashBox achieves competitive segmentation performance without the need for additional mask-specific training.

Methodology

The HashBox pipeline is composed of four distinct stages designed to bridge the gap between bounding box detection and pixel-level segmentation:

  1. Object Detection: The process begins with a standard object detector—specifically Faster R-CNN with a VGG-16 backbone—to generate candidate bounding boxes for objects within an image.
  2. Hierarchy Generation: In parallel, the image is decomposed into an Ultrametric Contour Map (UCM). This creates a hierarchical tree of regions, ranging from fine-grained superpixels to the entire image frame.
  3. Hierarchical Section Hashing (HSH): This core component utilizes Locality Sensitive Hashing (LSH) to efficiently match the patches within detected bounding boxes to the most relevant regions in the UCM hierarchy. By mapping similar visual features to the same hash buckets, the system can retrieve optimal masks in sublinear time.
  4. Hierarchical Section Pruning (HSP): A final refinement step resolves spatial overlaps between different object masks and filters out isolated noise, ensuring a clean and topologically consistent final segmentation.

Experimental Results

The authors evaluated HashBox on the PASCAL VOC 2012 segmentation benchmark. The method demonstrated that a "train-free" segmentation head could achieve results comparable to contemporary trained models.

Qualitative comparison between ground truth masks and the C&Z Segmentation (HashBox) results across various PASCAL VOC categories.

Key performance metrics on the validation set included an Instance-Level Jaccard Index of 45.24% and a Class-Level Jaccard Index of 43.05%. The system proved particularly effective for high-contrast objects, achieving Jaccard scores of 66.3% for "Cat" and 63.3% for "Dog."

Discussion and Future Work

The primary innovation of HashBox lies in its segmentation-as-retrieval philosophy. By assuming that high-quality hierarchies like UCM already contain the necessary object boundaries, the task simplifies to finding the correct node in the tree. This "plug-and-play" nature allows the segmentation logic to remain constant even as underlying bounding box detectors improve.

However, the method's performance is inherently bounded by two factors: the accuracy of the initial bounding box and the quality of the UCM hierarchy. If the hierarchy fails to capture a specific boundary, the retrieval mechanism cannot compensate. Future research directions include optimizing the Hierarchical Section Pruning (HSP) stage for dense scenes and extending these hashing-based retrieval techniques to 3D point cloud data, such as LIDAR for autonomous navigation.

Summary updated

Abstract

We propose a novel approach to address the Simultaneous Detection and Segmentation problem. Using hierarchical structures we use an efficient and accurate procedure that exploits the hierarchy feature information using Locality Sensitive Hashing. We build on recent work that utilizes convolutional neural networks to detect bounding boxes in an image and then use the top similar hierarchical region that best fits each bounding box after hashing, we call this approach iSegmentation. We then refine our final segmentation results by automatic hierarchy pruning. iSegmentation introduces a train-free alternative to Hypercolumns. We conduct extensive experiments on PASCAL VOC 2012 segmentation dataset, showing that iSegmentation gives competitive state-of-the-art object segmentations.
BibTeX
@misc{curto2017hashbox,
  author = {Joachim de Curtò and Irene Zarza and Alexander J. Smola and Luc Van Gool},
  title = {{HashBox: Hash Hierarchical Segmentation exploiting Bounding Box Object Detection}},
  year = {2017},
  eprint = {1702.08160},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1702.08160},
}
Figure