Object Detection via a Multi-region and Semantic Segmentation-Aware CNN Model

Spyros Gidaris, Nikos Komodakis

Abstract: We propose an object detection system that relies on a multi-region deep convolutional neural network (CNN) that also encodes semantic segmentation-aware features. The resulting CNN-based representation aims at capturing a diverse set of discriminative appearance factors and exhibits localization sensitivity that is essential for accurate object localization. We exploit the above properties of our recognition module by integrating it on an iterative localization mechanism that alternates between scoring a box proposal and refining its location with a deep CNN regression model. Thanks to the efficient use of our modules, we detect objects with very high localization accuracy. On the detection challenges of PASCAL VOC2007 and PASCAL VOC2012 we achieve mAP of 78.2% and 73.9% correspondingly, surpassing any other published work by a significant margin.

Introduction

In the wake of the revolutionary R-CNN (Regions with CNN features) framework, object detection research shifted toward deeper and more complex neural architectures. However, researchers Spyros Gidaris and Nikos Komodakis identified that simply increasing depth was insufficient for solving the core challenges of the field. Their work focuses on making representations "wider" and more "localization-aware."

By 2015, the standard detection pipeline faced two significant hurdles. First, CNNs were inherently designed for translation invariance to aid classification, which often conflicted with the need for precise bounding box placement. Second, standard models typically analyzed a single cropped box in isolation, struggling with occluded objects or scenarios where environmental context was vital for identification.

Methodology: The MR-CNN Framework

The proposed system, known as MR-CNN, introduces a sophisticated architecture designed to be explicitly sensitive to localization while leveraging both object parts and global context.

Multi-Region CNN Architecture

Rather than extracting features from a single bounding box, the model analyzes ten distinct regions for every candidate object. These regions include the original box, horizontal and vertical halves, central regions at different scales, the border area, and a large "context" region.

Overview of the MR-CNN architecture showing the activation maps and region adaptation modules.

As shown in the architecture diagram, the system utilizes an "Activation Maps Module" to process the entire image. A "Cropping Layer" then extracts features for various regions, which are passed into independent "Region Adaptation Modules." Each module consists of fully connected layers that learn to identify features specific to that spatial area (e.g., what a "top half" of a cow looks like versus the "border" transition between a cow and grass).

Defining the Ten Regions

The choice of regions is strategic, aiming to capture diverse appearance factors. By focusing on the transition between the object and the background (the Border Region), the model becomes significantly more sensitive to whether a bounding box is slightly misaligned.

The ten candidate regions used to capture diverse appearance factors and context.

Semantic Segmentation-Aware Features

To further enhance the representation, the authors integrated a branch that functions like a Fully Convolutional Network (FCN). This branch is trained to predict whether pixels are "foreground" (part of an object) or "background." Remarkably, this was achieved using only bounding box labels—a form of weak supervision. Despite the lack of pixel-perfect masks, the network learned to produce probability maps that helped the detector distinguish precise object boundaries.

Iterative Localization and Bounding Box Voting

Detection in MR-CNN is an iterative process rather than a single-shot classification. The model scores a box, uses a regression module to refine its location, and then re-scores the new, adjusted location. This is followed by a "voting" scheme where multiple high-scoring, overlapping boxes contribute to the final coordinates, resulting in much tighter and more accurate fits than standard non-maximum suppression.

Experimental Results

The model was rigorously evaluated on the PASCAL VOC 2007 and 2012 benchmarks, which were the primary standards for object detection at the time. The networks were pre-trained on ImageNet and subsequently fine-tuned on the VOC datasets.

  • PASCAL VOC 2007: The model achieved 78.2% mAP, a substantial improvement over the original R-CNN (~66%) and contemporary VGG-Net based models (~70%).
  • PASCAL VOC 2012: The model reached 73.9% mAP.
  • High-Precision Localization: When the success threshold (Intersection over Union) was raised to > 0.7, MR-CNN outperformed competitors by an even wider margin, validating its superior localization capabilities.

Insights and Discussion

A key finding of the study was the importance of borders; the "Border Region" proved incredibly effective at helping the model determine if a box was centered correctly. Additionally, the authors noted that while context is a powerful tool for identifying difficult objects (like a sheep on a mountainside), it can occasionally be misleading. The multi-region approach allows the model to weigh these contextual factors independently of the object's core features.

While highly accurate, the MR-CNN approach is computationally intensive due to the need to process ten regions per candidate box. Although the initial convolutional layers are shared, the width of the adaptation modules adds significant overhead. Consequently, the model was designed for maximum accuracy rather than real-time performance, setting a new benchmark for what was possible in high-precision object localization.

Summary updated

Abstract

We propose an object detection system that relies on a multi-region deep convolutional neural network (CNN) that also encodes semantic segmentation-aware features. The resulting CNN-based representation aims at capturing a diverse set of discriminative appearance factors and exhibits localization sensitivity that is essential for accurate object localization. We exploit the above properties of our recognition module by integrating it on an iterative localization mechanism that alternates between scoring a box proposal and refining its location with a deep CNN regression model. Thanks to the efficient use of our modules, we detect objects with very high localization accuracy. On the detection challenges of PASCAL VOC2007 and PASCAL VOC2012 we achieve mAP of 78.2% and 73.9% correspondingly, surpassing any other published work by a significant margin.
BibTeX
@inproceedings{gidaris2015object,
  author = {Spyros Gidaris and Nikos Komodakis},
  title = {{Object Detection via a Multi-region and Semantic Segmentation-Aware CNN Model}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=HyVkW-fO-B},
}
Figure