Learning to Segment Object Candidates via Recursive Neural Networks

Tianshui Chen, Liang Lin, Xian Wu, Nong Xiao, Xiaonan LuoIEEE TIP 2018 · DOI · OpenReview

TL;DR

This paper introduces a novel Recursive Neural Network (ReNN) approach for generating object proposals, which are bounding boxes likely to contain objects. The ReNN learns to group regions and predict objectness simultaneously, outperforming existing methods in recall and efficiency by adaptively learning similarity metrics and incorporating a randomized merging strategy to handle ambiguities.

What Problem Does This Paper Address?

Object detection systems heavily rely on generating a set of candidate "object proposals" – bounding boxes that are likely to contain objects. Traditional methods for generating these proposals often involve exhaustive search over locations and scales, or bottom-up region grouping with fixed similarity metrics. These approaches can be computationally expensive, sensitive to noise, and struggle with complex object shapes and occlusions, leading to missed objects or inaccurate boundaries. The paper aims to address these limitations by proposing a more adaptive and robust method for generating high-quality object proposals.

Methods

The core of the proposed method is a Recursive Neural Network (ReNN) that processes an image hierarchically. The framework consists of four main modules:

  • Local Feature Extraction: Uses a pre-trained Fast R-CNN architecture (VGG16-net) to extract deep local features for each initial region (obtained via graph-based over-segmentation).
  • Semantic Mapper: Maps local region features into a semantic space.
  • Feature Combiner: Aggregates semantic features from neighboring child regions to create a semantic representation for parent regions.
  • Merging Scorer: Predicts a score indicating the confidence that two neighboring regions should be merged.
  • Objectness Scorer: Predicts an "objectness" score for each potential proposal, indicating the likelihood of it containing an object.

A key innovation is the joint optimization of the region merging similarity and the objectness prediction within the ReNN. This is achieved through a structured loss function that penalizes incorrect region groupings.

To address the sensitivity of greedy merging algorithms to noise and local minima, the paper introduces a Randomized Merging Algorithm. Instead of always merging the pair with the highest similarity score, it selects one pair from the top-k highest similarity pairs based on a probability distribution derived from their scores. This process is repeated multiple times to increase the diversity of proposals and improve recall.

Datasets

The paper uses the following datasets for training and evaluation:

  • PASCAL VOC 2007: Used for training and testing.
  • PASCAL VOC 2012: Used for validation.
  • ImageNet 2015 validation set: Used to evaluate generalization to unseen object categories.

Key Insights and Ideas

  • Adaptive Similarity Learning: Instead of relying on hand-crafted similarity metrics, learning these metrics directly within the ReNN allows the model to adapt to different image characteristics and object types.
  • End-to-End Joint Optimization: Simultaneously learning region merging and objectness prediction within a single framework leads to more coherent and accurate proposals.
  • Randomized Merging for Robustness: Introducing randomness into the merging process significantly improves recall by mitigating the risk of irreversible incorrect merges that can occur with greedy approaches. This is particularly effective for objects that are easily confused with the background.
  • Hierarchical Structure Mirrors Bottom-Up Grouping: The ReNN's recursive structure naturally aligns with traditional bottom-up region grouping pipelines, making it an intuitive integration.

Key Results

  • Superior Recall and Accuracy: The proposed ReNN method achieves state-of-the-art performance on PASCAL VOC 2007 and 2012 benchmarks, particularly in terms of recall rates at stricter IoU thresholds (e.g., 0.8).
  • Improved Efficiency: The method is significantly faster than some comparable region-grouping methods (e.g., 7x faster than MCG2015) while maintaining high accuracy. The feature extraction takes ~0.2s, and one-time random merging takes ~0.5s.
  • Generalization: The model demonstrates good generalization to unseen object categories on the ImageNet dataset, outperforming methods like RPN.
  • Object Detection Performance: When used with Fast R-CNN, the proposed proposals lead to state-of-the-art mean Average Precision (mAP) on PASCAL VOC 2007.
  • Randomized Merging Benefits: The randomized merging strategy consistently improves recall, especially at higher IoU thresholds, with diminishing returns after a certain number of random iterations.

Limitations and Future Work

  • Small Object Recall: The paper acknowledges that the method performs slightly worse than RPN for very small objects (area < 5k pixels), potentially due to the reliance on over-segmentation accuracy.
  • Future Work: The authors suggest that the proposed framework can be tightly combined with category-specific object detection methods for further improvements. They also hint at exploring more advanced ways to integrate the randomized merging process for even greater stability and recall.

Summary updated

Abstract

To avoid the exhaustive search over locations and scales, current state-of-the-art object detection systems usually involve a crucial component generating a batch of candidate object proposals from images. In this paper, we present a simple yet effective approach for segmenting object proposals via a deep architecture of recursive neural networks (ReNNs), which hierarchically groups regions for detecting object candidates over scales. Unlike traditional methods that mainly adopt fixed similarity measures for merging regions or finding object proposals, our approach adaptively learns the region merging similarity and the objectness measure during the process of hierarchical region grouping. Specifically, guided by a structured loss, the ReNN model jointly optimizes the cross-region similarity metric with the region merging process as well as the objectness prediction. During inference of the object proposal generation, we introduce randomness into the greedy search to cope with the ambiguity of grouping regions. Extensive experiments on standard benchmarks, e.g., PASCAL VOC and ImageNet, suggest that our approach is capable of producing object proposals with high recall while well preserving the object boundaries and outperforms other existing methods in both accuracy and efficiency.
BibTeX
@article{chen2018learning,
  author = {Tianshui Chen and Liang Lin and Xian Wu and Nong Xiao and Xiaonan Luo},
  title = {{Learning to Segment Object Candidates via Recursive Neural Networks}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  doi = {10.1109/tip.2018.2859025},
}
Figure