Object detection via a multi-region & semantic segmentation-aware CNN model

Spyros Gidaris, Nikos Komodakis

Abstract

We propose an object detection system that relies on a multi-region deep convolutional network (CNN) that also encodes semantic segmentation-aware features. The resulting CNN-based representation aims at capturing a diverse set of discriminative appearance factors and exhibits localization sensitivity that is essential for accurate object localization. We exploit the above properties of our recognition module by integrating it on an iterative localization mechanism that alternates between scoring a box proposal and refining its location with a deep CNN regression model. Thanks to the efficient use of our modules, we detect objects with very high localization accuracy. On the detection challenges of PASCAL VOC2007 and PASCAL VOC2012 we achieve mAP of 78.2% and 73.9% correspondingly, surpassing any other published work by a significant margin.

Introduction

In 2015, the "R-CNN" paradigm had revolutionized object detection, but it faced two major hurdles: representation and localization. Standard CNNs are designed to be translation-invariant, which is beneficial for classification but detrimental for precise localization. Most models treated objects as monolithic blocks, often missing crucial context or failing under partial occlusion. This research identifies that the bottleneck for high-performance detection is not just classification accuracy, but the ability to precisely delineate object boundaries.

Methodology

The proposed framework introduces a three-pronged approach to improve localization accuracy:

Multi-Region CNN

Instead of extracting features from a single bounding box, the model analyzes 10 different sub-regions for every candidate box. These include the original box, half-boxes (left, right, top, bottom), central regions, and rectangular "border regions." This architecture forces the network to learn about object parts, edges, and immediate surrounding context.

Semantic Segmentation-Aware Features

A parallel branch in the network learns to predict foreground/background masks. This is achieved through weak supervision, using existing bounding box annotations to estimate object presence without requiring pixel-level labels. This helps the model better understand object boundaries.

Iterative Localization & Bounding Box Voting

The system employs an iterative process where it scores a box, refines its coordinates via a CNN regression model, and then re-scores the new box. A final "voting" scheme aggregates multiple high-scoring, overlapping boxes to determine the final location.

The proposed multi-region CNN architecture, featuring activation maps, cropping layers, and parallel region adaptation modules.

Results and Analysis

The model was evaluated on the PASCAL VOC 2007 and 2012 benchmarks, using VGG-Net as the backbone.

  • PASCAL VOC 2007: Achieved a Mean Average Precision (mAP) of 78.2%, a 12.2-point improvement over the original R-CNN.
  • PASCAL VOC 2012: Achieved 73.9% mAP, setting a new state-of-the-art at the time of publication.
Qualitative detection results on various classes from the PASCAL VOC dataset, demonstrating precise localization.

The "Detection Error Analysis" revealed a significant reduction in false positives caused by poor localization. By focusing on border regions, the model intentionally breaks translation invariance to become more sensitive to box fit.

Discussion and Limitations

While the multi-region approach significantly boosts accuracy, it introduces higher computational overhead compared to emerging single-shot detectors. The model still faces challenges with adjacent or overlapping objects of the same class, such as huddles of sheep or rows of bicycles. Furthermore, qualitative analysis suggests the model occasionally detects objects correctly that were omitted in the ground-truth annotations, indicating it may be approaching the limits of the benchmarks themselves.

Summary updated

Abstract

We propose an object detection system that relies on a multi-region deep convolutional neural network (CNN) that also encodes semantic segmentation-aware features. The resulting CNN-based representation aims at capturing a diverse set of discriminative appearance factors and exhibits localization sensitivity that is essential for accurate object localization. We exploit the above properties of our recognition module by integrating it on an iterative localization mechanism that alternates between scoring a box proposal and refining its location with a deep CNN regression model. Thanks to the efficient use of our modules, we detect objects with very high localization accuracy. On the detection challenges of PASCAL VOC2007 and PASCAL VOC2012 we achieve mAP of 78.2% and 73.9% correspondingly, surpassing any other published work by a significant margin.
BibTeX
@inproceedings{gidaris2015object,
  author = {Spyros Gidaris and Nikos Komodakis},
  title = {{Object detection via a multi-region \& semantic segmentation-aware CNN model}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=HyVkW-fO-B},
}
Figure