CityPersons: A Diverse Dataset for Pedestrian Detection

Shanshan Zhang, Rodrigo Benenson, Bernt Schiele

Abstract

Pedestrian detection is a cornerstone of autonomous driving and surveillance, yet progress has been hampered by data limitations and increasing architectural complexity. Existing benchmarks like Caltech-USA and KITTI are low-density and lack geographic diversity. We introduce CityPersons, a new, high-quality pedestrian detection dataset built on top of Cityscapes, featuring diverse urban environments from 27 different cities. We demonstrate that a properly tuned version of the standard Faster R-CNN architecture can outperform complex custom models, achieving state-of-the-art results and superior generalization across multiple benchmarks.

1. Introduction

Pedestrian detection models often struggle to generalize to the "open world" due to the limited diversity of training data. Traditional benchmarks like Caltech-USA average only one person per image and were recorded in a single city. Furthermore, recent research trends have favored highly customized, complex CNN architectures, potentially overlooking the untapped potential of standard detectors. This work addresses these gaps by providing a more diverse dataset and optimizing a "vanilla" Faster R-CNN for the task.

Overview of the CityPersons pipeline showing how diverse training data leads to a single model that generalizes across multiple benchmarks.

2. The CityPersons Dataset

CityPersons provides 5,000 images with approximately 35,000 person annotations across 27 cities. It features a high person density (~7 per image) and diverse weather conditions and seasons.

A key contribution is the use of "amodal" bounding box annotations. Unlike "visible" boxes that only encompass the seen parts of a person, amodal boxes cover the entire body extent even if partially occluded. This helps the detector learn the true proportions of a human body and improves alignment through a fixed aspect ratio (0.41).

Comparison of visible (BB-vis) and amodal (BB-full) bounding boxes for an occluded pedestrian.

3. Methodology: Adapting Faster R-CNN

The authors propose five specific modifications to make a standard Faster R-CNN suitable for pedestrian detection:

  • Quantized RPN scales: Adjusting the Region Proposal Network based on training data statistics.
  • Input up-scaling: Increasing resolution by 2x to match ImageNet pre-training distributions.
  • Finer feature stride: Reducing the stride from 16px to 8px to preserve resolution for small objects.
  • Ignore region handling: Preventing the model from treating ambiguous areas (like crowds) as background.
  • Adam Solver: Utilizing the Adam optimizer for more consistent training gains.

Additionally, semantic segmentation maps were used as extra input channels to provide environmental context, which significantly aided in detecting small pedestrians where visual features are often blurry.

4. Key Results and Generalization

The adapted Faster R-CNN achieved a massive improvement on the Caltech-USA benchmark, reducing the Log Miss-Rate (MR) from 20.98% to 10.27%.

More importantly, models trained on CityPersons demonstrated superior generalization. When tested on unseen datasets like ETH or KITTI without fine-tuning, the CityPersons-trained model consistently outperformed those trained on Caltech. For instance, on the ETH dataset, the CityPersons model achieved a 35.64% MR, compared to 57.85% for the Caltech-trained model. This suggests that the variety of the training environment is as important as the quantity of data.

5. Conclusion

The success of CityPersons highlights that diversity is king for real-world deployment. Furthermore, the results serve as a reminder that well-tuned "plain" models can match or beat custom-designed architectures, suggesting that architectural complexity often compensates for sub-optimal data or configuration. Future work may exploit additional Cityscapes modalities, such as stereo imagery and video sequences, to further improve robustness.

Summary updated

BibTeX
@inproceedings{zhang2017citypersons,
  author = {Shanshan Zhang and Rodrigo Benenson and Bernt Schiele},
  title = {{CityPersons: A Diverse Dataset for Pedestrian Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2017},
  doi = {10.1109/cvpr.2017.474},
}
Figure