Improving Object Detection with Region Similarity Learning

Feng Gao, Yihang Lou, Yan Bai, Shiqi Wang, Tiejun Huang, Ling-Yu Duan

Abstract

Modern region-based object detectors often struggle to distinguish between positive object regions and "hard negative" background regions that contain partial object features. This paper proposes a novel approach to improve detection performance by explicitly learning region similarity. By introducing a class-specific triplet embedding loss, the network is trained to pull positive regions closer to ground truth while pushing hard negatives further away in the feature space. Combined with a "Top-K pooling" strategy to mitigate feature noise, this method achieves significant mean Average Precision (mAP) improvements on PASCAL VOC benchmarks and is highly compatible with existing architectures like Fast R-CNN and OHEM.

Introduction

Object detection frameworks typically classify region proposals (RoIs) into discrete categories or background. A persistent challenge is the presence of hard negatives—background regions with an Intersection over Union (IoU) less than 0.5 that nonetheless contain confusing visual elements from the object. Standard classification losses, such as cross-entropy, do not explicitly optimize the relative distances between these features. This paper addresses this gap by forcing the network to learn a structured embedding space where object regions are geometrically distinct from background clutter.

Methodology

Class-Specific Triplet Embedding

The core contribution is the addition of a triplet loss alongside standard classification and bounding-box regression objectives. To make the triplet loss effective for detection, the authors assign a "pseudo-class" to background RoIs based on their highest-overlapping ground-truth object.

Illustration of the triplet embedding space where positive regions ($R^p$) are pulled toward the ground truth ($R^g$) and hard negative regions ($R^n$) are pushed away by a margin $\alpha$.

The loss penalizes the model if the L2 distance between a ground-truth object ($R^g$) and a positive RoI ($R^p$) is not significantly smaller than the distance between $R^g$ and a hard negative RoI ($R^n$) of the same pseudo-class. This explicit margin $\alpha$ ensures that the classifier can more easily separate objects from visually similar background patches.

Hard Triplet Sampling and Top-K Pooling

To manage the computational complexity of triplet generation, the authors employ an active sampling strategy. Within each class group, they select only the "hardest" triplets: those where positive RoIs are furthest from the ground truth and negative RoIs are closest to it.

Furthermore, the paper introduces Top-K Pooling. Standard max pooling can be overly sensitive to noise in deep layers, while mean pooling may dilute important signals. Top-K pooling calculates the mean of the $K$ highest activations within a window, providing a robust statistical representation that ignores low-level background noise.

Overview of the region similarity learning framework, showing object detection (a), distance comparisons between regions (b), and mined positive/negative examples for specific classes (c).

Experimental Results

The proposed method was evaluated on the PASCAL VOC 2007 and 2012 datasets.

  • Baseline Improvements: Adding triplet embedding and Top-K pooling to a Fast R-CNN (VGG16) baseline increased mAP from 66.9% to 68.7% on VOC 2007.
  • Data Scaling: Training on combined VOC 07+12 data yielded a 2.1% mAP boost (70.0% to 72.1%).
  • Complementarity with OHEM: When integrated with Online Hard Example Mining (OHEM), the method provided an additional 1.2% mAP gain, reaching 75.8% on VOC 07+12.
  • Qualitative Impact: The approach significantly reduced false positives for visually ambiguous classes. For example, performance on "chairs" improved by up to 10% mAP.

Conclusion

By explicitly optimizing the feature distances between positive and negative regions, this work enhances the discriminative power of region-based detectors. While the primary gains are in classification rather than localization, the modular nature of triplet embedding and Top-K pooling makes them valuable additions to modern detection pipelines, particularly for reducing background-related errors.

Summary updated

Abstract

Object detection aims to identify instances of semantic objects of a certain class in images or videos. The success of state-of-the-art approaches is attributed to the significant progress of object proposal and convolutional neural networks (CNNs). Most promising detectors involve multi-task learning with an optimization objective of softmax loss and regression loss. The first is for multi-class categorization, while the latter is for improving localization accuracy. However, few of them attempt to further investigate the hardness of distinguishing different sorts of distracting background regions (i.e., negatives) from true object regions (i.e., positives). To improve the performance of classifying positive object regions vs. a variety of negative background regions, we propose to incorporate triplet embedding into learning objective. The triplet units are formed by assigning each negative region to a meaningful object class and establishing class-specific negatives, followed by triplets construction. Over the benchmark PASCAL VOC 2007, the proposed triplet embedding has improved the performance of well-known Fas-tRCNN model with a mAP gain of 2.1%. In particular, the state-of-the-art approach OHEM can benefit from the triplet embedding and has achieved a mAP improvement of 1.2%.
BibTeX
@inproceedings{gao2017improving,
  author = {Feng Gao and Yihang Lou and Yan Bai and Shiqi Wang and Tiejun Huang and Ling-Yu Duan},
  title = {{Improving Object Detection with Region Similarity Learning}},
  booktitle = {ICME},
  year = {2017},
  doi = {10.1109/icme.2017.8019372},
}
Figure