Abstract
We propose a novel approach to address the Simultaneous Detection and Segmentation problem. Using hierarchical structures we use an efficient and accurate procedure that exploits the hierarchy feature information using Locality Sensitive Hashing. We build on recent work that utilizes convolutional neural networks to detect bounding boxes in an image and then use the top similar hierarchical region that best fits each bounding box after hashing, we call this approach iSegmentation. We then refine our final segmentation results by automatic hierarchy pruning. iSegmentation introduces a train-free alternative to Hypercolumns. We conduct extensive experiments on PASCAL VOC 2012 segmentation dataset, showing that iSegmentation gives competitive state-of-the-art object segmentations.
Introduction
Simultaneous Detection and Segmentation (SDS) is a foundational challenge in computer vision, requiring a system to both localize objects with bounding boxes and delineate their precise pixel-level boundaries. While modern deep learning architectures have achieved remarkable success in this domain, they typically rely on end-to-end models that require extensive labeled data and significant training resources.
The "Segmentation of Objects by Hashing" (or iSegmentation) approach offers a modular alternative. By decoupling the detection and segmentation phases, the authors demonstrate that high-quality object masks can be generated without a dedicated training step for the segmentation component itself. Instead, the method leverages classical data structures and hierarchical image representations to "retrieve" the best mask for a given detection.
Methodology
The iSegmentation pipeline consists of four distinct stages designed to move from raw pixels to instance-level masks:
- Detection: A standard object detector (Faster R-CNN with a VGG-16 backbone) identifies objects and generates bounding boxes.
- Hierarchy Generation: The image is decomposed into a tree of hierarchical regions using an Ultrametric Contour Map (UCM). This tree captures potential object boundaries across multiple scales.
- Hierarchical Section Hashing (HSH): This core innovation utilizes Locality Sensitive Hashing (LSH) to index the UCM regions. For every detected bounding box, the system performs a fast lookup in the hash map to find the hierarchical region that most closely matches the box's spatial support.
- Hierarchical Section Pruning (HSP): A final refinement step handles overlapping masks. By prioritizing higher-level regions that better encompass objects and removing isolated noise, the system produces a clean final segmentation.
Crucially, this process is train-free. Once the detector is trained, the segmentation logic relies entirely on the geometric properties of the image hierarchy and efficient retrieval.
Experimental Results
The authors evaluated iSegmentation on the PASCAL VOC 2012 segmentation benchmark. Performance was measured using the Jaccard Index (Intersection over Union, IoU).
The method achieved a global Jaccard index of 45.24% for instance-level accuracy and 43.05% for class-level accuracy. The system proved particularly effective for high-contrast objects, such as cats (66.3% IoU) and dogs (63.3% IoU). These results are competitive with established methods like Hypercolumns, despite the significantly reduced complexity of the segmentation phase.
Discussion and Limitations
The modularity of iSegmentation allows for easy upgrades; for instance, the bounding box detector could be swapped for a more advanced model without retraining the segmentation logic. However, the system's performance is inherently capped by the detector's accuracy. If an object is missed during the detection phase, the hashing mechanism cannot generate a mask. Additionally, while the hashing lookup is efficient, the initial generation of high-quality UCMs remains a computational bottleneck.
Future work may explore linking these retrieval-based concepts with Kernel Methods to create a more unified theoretical framework bridging deep learning and similarity-based retrieval.
Summary updated