JHU-CROWD++: Large-Scale Crowd Counting Dataset and A Benchmark Method

Vishwanath A. Sindagi, Rajeev Yasarla, Vishal M. PatelIEEE TPAMI 2020 · OpenReview

TL;DR

This paper introduces JHU-CROWD++, a new large-scale dataset for crowd counting that addresses limitations of existing datasets by including diverse scenarios, adverse weather conditions, and richer annotations. They also propose CG-DRCN, a novel network that progressively refines crowd density maps using uncertainty-guided residual learning, achieving improved accuracy.

What Problem Does This Paper Address?

Accurate crowd counting is crucial for applications like public safety, urban planning, and traffic management. While deep learning has significantly advanced this field, existing datasets often suffer from limited diversity, insufficient annotations, and biases that hinder the development of robust models. Specifically, many datasets lack images captured under challenging environmental conditions (e.g., rain, fog, haze) and don't provide detailed annotations beyond simple head locations. This makes it difficult for models to generalize to real-world scenarios. The authors aim to tackle these issues by creating a more comprehensive dataset and developing a network that can better handle these complexities.

Methods

The core of the proposed method is the Confidence Guided Deep Residual Counting Network (CG-DRCN). It builds upon a VGG16 backbone and employs a progressive refinement strategy.

  • Progressive Density Map Generation: Instead of directly predicting the final density map, the network generates a coarse map from a deeper layer and then refines it using residual learning from shallower layers. This allows for capturing both global context and fine-grained details.
  • Uncertainty-Guided Residual Learning (U-REB): To ensure that only reliable information is used for refinement, an "Uncertainty Guided Residual Estimation Block" (U-REB) is introduced. This block estimates the confidence of the residual errors and gates them, propagating only high-confidence residuals. This is inspired by uncertainty estimation in CNNs.
  • Class-Conditioned Refinement (U-REBC): To leverage the new weather annotations in their dataset, the U-REB is augmented with a "Class Conditioning" (CC) block. This allows the network to condition the residual estimation on image-level weather labels (rain, fog, haze, normal), further improving performance in adverse conditions.
  • Loss Function: The loss function combines a regression loss for density map accuracy with a confidence guiding loss and a cross-entropy loss for weather classification.

Datasets

  • JHU-CROWD++ (New Dataset): This is the primary contribution. It contains 4,372 images with 1.51 million head annotations. Key features include:
  • Diverse scenarios and environmental conditions.
  • Images with weather-based degradations (rain, snow, fog, haze).
  • Distractor images to mitigate dataset bias.
  • Rich annotations: point-wise locations, occlusion level, blur level, approximate head size, and image-level labels (scene type, weather).
  • Existing Datasets: The paper also evaluates their method on:
  • ShanghaiTech Dataset [5]
    • UCF-QNRF Dataset [21]

Key Insights and Ideas

  • Richer Annotations are Crucial: The detailed annotations in JHU-CROWD++ (beyond just head locations) are shown to be vital for improving model robustness, particularly for handling adverse weather conditions.
  • Progressive Refinement with Residuals: Directly predicting dense maps can be challenging. The paper demonstrates that a progressive refinement approach, where errors from deeper layers are corrected by shallower ones, leads to better results.
  • Uncertainty as a Gating Mechanism: Using uncertainty estimation to filter residuals is a powerful way to prevent noisy or unreliable information from degrading the final prediction. This highlights the importance of understanding model confidence.
  • Leveraging Image-Level Labels: Conditioning the network on image-level weather information significantly boosts performance in challenging environmental conditions, showing the benefit of multi-task learning or conditional processing.
  • Generalizability of the Approach: The proposed CG-DRCN architecture, particularly with the uncertainty and class-conditioning, shows good performance not only with the VGG16 backbone but also with a ResNet-101 backbone, indicating its generalizability.

Key Results

  • JHU-CROWD++ Performance: The proposed CG-DRCN-CC-Res101 achieves the lowest overall MAE on the JHU-CROWD++ validation set, outperforming other methods in "high-density" and "weather" categories.
  • ShanghaiTech Dataset: The proposed method outperforms existing methods on Part A of the ShanghaiTech dataset and achieves comparable performance on Part B.
  • UCF-QNRF Dataset: The CG-DRCN-Res101 achieves comparable performance to state-of-the-art methods on UCF-QNRF.
  • Ablation Studies:
    • Residual learning alone improves performance.
  • Adding uncertainty-guided residual estimation (U-REB) further boosts accuracy.
  • Class-conditioning (U-REBC) provides significant improvements, especially for weather conditions.
  • Using more residual branches (e.g., from conv5, conv4, conv3) leads to progressively lower errors.
  • The model benefits significantly from larger training datasets, with MAE decreasing as more data is used.

Limitations and Future Work

  • Dataset Size vs. Complexity: While JHU-CROWD++ is large, the authors acknowledge that the complexity of real-world crowd scenes might still pose challenges.
  • Annotation Effort: The paper mentions that obtaining detailed annotations like size is labor-intensive, hinting at potential scalability issues for even larger datasets with such rich annotations.
  • Future Work: The paper suggests exploring other backbone networks and further investigating the impact of different annotation types. The generalizability of the proposed method to other computer vision tasks that involve density estimation or handling adverse conditions could also be explored.

Summary updated

Abstract

We introduce a new large scale unconstrained crowd counting dataset (JHU-CROWD++) that contains “4,372” images with “1.51 million” annotations. In comparison to existing datasets, the proposed dataset is collected under a variety of diverse scenarios and environmental conditions. Specifically, the dataset includes several images with weather-based degradations and illumination variations, making it a very challenging dataset. Additionally, the dataset consists of a rich set of annotations at both image-level and head-level. Several recent methods are evaluated and compared on this dataset. The dataset can be downloaded from <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><uri>http://www.crowd-counting.com</uri></i> . Furthermore, we propose a novel crowd counting network that progressively generates crowd density maps via residual error estimation. The proposed method uses VGG16 as the backbone network and employs density map generated by the final layer as a coarse prediction to refine and generate finer density maps in a progressive fashion using residual learning. Additionally, the residual learning is guided by an uncertainty-based confidence weighting mechanism that permits the flow of only high-confidence residuals in the refinement path. The proposed Confidence Guided Deep Residual Counting Network (CG-DRCN) is evaluated on recent complex datasets, and it achieves significant improvements In errors.
BibTeX
@article{sindagi2020jhu,
  author = {Vishwanath A. Sindagi and Rajeev Yasarla and Vishal M. Patel},
  title = {{JHU-CROWD++: Large-Scale Crowd Counting Dataset and A Benchmark Method}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2020},
  url = {https://openreview.net/forum?id=F6rgNgaIcqzA},
}
Figure