CrowdNet: A Deep Convolutional Network for Dense Crowd Counting

Lokesh Boominathan, Srinivas S. S. Kruthiventi, R. Venkatesh Babu

Abstract

Our work proposes a novel deep learning framework for estimating crowd density from static images of highly dense crowds. We use a combination of deep and shallow, fully convolutional networks to predict the density map for a given crowd image. Such a combination is used for effectively capturing both the high-level semantic information (face/body detectors) and the low-level features (blob detectors), that are necessary for crowd counting under large scale variations. As most crowd datasets have limited training samples (<100 images) and deep learning based approaches require large amounts of training data, we perform multi-scale data augmentation. Augmenting the training samples in such a manner helps in guiding the CNN to learn scale invariant representations. Our method is tested on the challenging UCF_CC_50 dataset, and shown to outperform the state of the art methods.

Introduction

Crowd counting is a critical task for public safety and urban planning, yet it is notoriously difficult in highly dense scenarios where a single frame may contain thousands of individuals. Traditional computer vision techniques, such as face or body detectors, often fail because people are heavily occluded and appear only as small clusters of pixels. Furthermore, these images suffer from extreme perspective distortion, where the scale of individuals varies significantly based on their distance from the camera.

Another major hurdle is the scarcity of annotated data. Manually labeling thousands of heads in a single image is a labor-intensive task, leaving researchers with very small datasets (often fewer than 100 images) to train data-hungry deep learning models. CrowdNet addresses these challenges through a dual-stream architecture and robust data augmentation strategies.

Methodology

The authors introduce CrowdNet, a dual-column fully convolutional network (FCN) architecture designed to capture features at multiple scales:

  1. The Deep Network: Based on the VGG-16 architecture, this branch captures high-level semantic information to identify "person-like" features. To maintain high-resolution output, the final pooling layers were removed and replaced with dilated convolutions (convolutions with holes), allowing the network to maintain a large receptive field without losing spatial detail.
  2. The Shallow Network: This branch consists of three convolutional layers with large filters ($5 \times 5$). It is specifically designed to detect low-level patterns, such as the small "blobs" that represent heads in the distance.
  3. Density Map Estimation: Instead of regressing a single integer, the network predicts a spatial "density map." Ground-truth annotations (dots) are blurred with a Gaussian kernel for training. The final count is obtained by integrating (summing) the predicted map.
The CrowdNet architecture combines a deep VGG-based network with a shallow network to capture multi-scale features.

To combat small dataset sizes and scale variation, the team employed multi-scale augmentation. Patches were cropped from images at multiple scales (0.5x to 1.2x), and high-density regions were over-sampled to help the model learn to handle the most difficult cases.

Results and Discussion

CrowdNet was evaluated primarily on the UCF_CC_50 dataset, which contains 50 images with counts ranging from 94 to over 4,500 individuals. The model outperformed previous state-of-the-art methods, achieving a Mean Absolute Error (MAE) of 452.5, compared to the previous best of 467.0.

Qualitative results comparing actual ground truth counts with CrowdNet's estimated counts and density maps.

An ablation study confirmed the necessity of the dual-stream approach. On the most difficult test fold, the combined CrowdNet (645 MAE) significantly outperformed the Deep Network (681 MAE) or Shallow Network (1107 MAE) alone. Furthermore, density-based data augmentation improved the MAE from 725 to 645 on these difficult cases, proving that multi-scale training helps the CNN learn scale-invariant representations.

Conclusion

CrowdNet demonstrates that combining deep semantic features with shallow blob detection is an effective strategy for dense crowd counting. While the model still tends to underestimate counts in extreme scenarios (over 2,500 people) due to training data limitations, it sets a new benchmark for accuracy. Future work may involve incorporating temporal information from video feeds or utilizing perspective maps to further refine density estimation.

Summary updated

Abstract

Our work proposes a novel deep learning framework for estimating crowd density from static images of highly dense crowds. We use a combination of deep and shallow, fully convolutional networks to predict the density map for a given crowd image. Such a combination is used for effectively capturing both the high-level semantic information (face/body detectors) and the low-level features (blob detectors), that are necessary for crowd counting under large scale variations. As most crowd datasets have limited training samples (<100 images) and deep learning based approaches require large amounts of training data, we perform multi-scale data augmentation. Augmenting the training samples in such a manner helps in guiding the CNN to learn scale invariant representations. Our method is tested on the challenging UCF_CC_50 dataset, and shown to outperform the state of the art methods.
BibTeX
@inproceedings{boominathan2016crowdnet,
  author = {Lokesh Boominathan and Srinivas S. S. Kruthiventi and R. Venkatesh Babu},
  title = {{CrowdNet: A Deep Convolutional Network for Dense Crowd Counting}},
  booktitle = {ACM Multimedia},
  year = {2016},
  url = {https://openreview.net/forum?id=zKweXatZxo},
}
Figure