Is Faster R-CNN Doing Well for Pedestrian Detection?

Liliang Zhang, Liang Lin, Xiaodan Liang, Kaiming He

Abstract

While Faster R-CNN has established itself as a leading framework for general object detection, its performance in pedestrian detection initially lagged behind hybrid methods utilizing hand-crafted features. This paper investigates the root causes of this performance gap, identifying low feature resolution and the absence of effective hard-negative mining as primary obstacles. By integrating a Region Proposal Network (RPN) with a Boosted Forest (BF) classifier and employing high-resolution convolutional features, the authors demonstrate that a purely deep-learning approach can achieve state-of-the-art results on major pedestrian benchmarks.

Introduction

In 2016, the computer vision community faced a paradox: Faster R-CNN was the "gold standard" for general object detection (e.g., COCO, PASCAL VOC), yet it was surprisingly mediocre at detecting pedestrians. Most top-performing pedestrian detectors at the time still relied on hybrid models—combining traditional hand-crafted features like HOG or Integral Channel Features with deep learning. This research, a collaboration between Sun Yat-sen University and Microsoft Research, sought to bridge this gap by refining the Faster R-CNN pipeline specifically for the unique challenges of pedestrian detection.

Identifying the "Blind Spots"

The authors pinpoint two specific weaknesses in the standard Faster R-CNN pipeline that hinder its effectiveness for pedestrians:

  1. Resolution Mismatch: Pedestrians in surveillance or automotive datasets are often very small (e.g., 28x70 pixels). Standard RoI pooling on a feature map with a 16-pixel stride causes "bin collapsing," where the spatial information of a small person is squashed into a single, non-discriminative point.
  2. Background Confusion: Unlike general object detection, where the challenge is distinguishing between classes (e.g., dog vs. cat), pedestrian detection primarily involves distinguishing a person from "hard" background clutter, such as trees or poles. Standard Faster R-CNN lacks a dedicated mechanism for "bootstrapping"—the iterative process of finding and training on these difficult false positives.
Examples of small positive pedestrian instances (left) and hard negatives (right) that challenge standard detectors.

Proposed Methodology: RPN + BF

The authors propose a simplified but highly effective pipeline that replaces the standard downstream classifier with a Boosted Forest (BF).

Tailored RPN and High-Resolution Features

The Region Proposal Network (RPN) is adapted specifically for human proportions. Instead of multiple aspect ratios, the model uses a single ratio (0.41, the average for pedestrians) across nine different scales. To address the resolution issue, the "à trous" algorithm (dilated convolutions) is used to increase feature map resolution without retraining. Features are pooled from shallower, higher-resolution layers (like Conv3_3 and Conv4_3) to preserve spatial detail.

Boosted Forest and Bootstrapping

Instead of the standard Multi-Layer Perceptron (MLP), RPN proposals are fed into a cascaded Boosted Forest. This classifier is trained in six stages. After each stage, the model "mines" the dataset for hard negatives—background areas incorrectly labeled as people—and adds them to the training set for the next stage. This forces the model to learn the subtle differences between a person and person-shaped environmental clutter.

The proposed pedestrian detection pipeline: RPN generates proposals which are then classified by a Boosted Forest.

Experimental Results

The method was evaluated on four major benchmarks: Caltech, INRIA, ETH, and KITTI.

  • Caltech: The model achieved a 9.6% Miss Rate (MR), a significant improvement over the previous best of 11.7%. It also showed a 40% relative improvement in localization accuracy at a strict IoU threshold of 0.7.
  • KITTI: The system ranked at the top of the leaderboard for "Easy" pedestrians (77.12% mAP) and remained highly competitive in "Moderate" and "Hard" categories.
  • Efficiency: Despite the added complexity of dilated convolutions, the system runs at 0.5 seconds per image on a Tesla K40 GPU.

Conclusion

This work proved that deep features contain all the necessary information for high-accuracy pedestrian detection, provided they are extracted at the correct resolution and trained with an effective mining strategy. By identifying that a properly tuned RPN is a powerful detector on its own, and that bootstrapping is a critical strategy for handling background confusion, the authors effectively ended the era of hand-crafted features in the field.

Summary updated

BibTeX
@inproceedings{zhang2016is,
  author = {Liliang Zhang and Liang Lin and Xiaodan Liang and Kaiming He},
  title = {{Is Faster R-CNN Doing Well for Pedestrian Detection?}},
  booktitle = {European Conference on Computer Vision},
  year = {2016},
  doi = {10.1007/978-3-319-46475-6\_28},
}
Figure