Evaluating Crowd Density Estimators via Their Uncertainty Bounds

Jennifer Vandoni, Emanuel Aldea, Sylvie Le Hégarat-Mascle

Abstract

In this work, we use the Belief Function Theory which extends the probabilistic framework in order to provide uncertainty bounds to different categories of crowd density estimators. Our method allows us to compare the multi-scale performance of the estimators, and also to characterize their reliability for crowd monitoring applications requiring varying degrees of prudence.

Introduction

Crowd density estimation is a critical component of urban planning and public safety management. While deep learning models have achieved impressive accuracy in terms of point estimates, they often lack a measure of confidence. Standard metrics like Mean Absolute Error (MAE) can mask local inaccuracies, as overestimations in one region may cancel out underestimations in another. In high-stakes environments, such as stadium management or religious pilgrimages, understanding the reliability of a count is as important as the count itself.

Methodology

The FE+LFE Architecture

To address the challenges of high-density crowd counting, we propose a fully convolutional network architecture. It consists of a Front End (FE) that utilizes increasing dilation factors to capture broad context without sacrificing resolution, and a Local Feature Extractor (LFE) with decreasing dilation factors to maintain spatial consistency. This design is optimized for small target detection in cluttered scenes while remaining compact to avoid overfitting on limited datasets.

Evidential CNN-Ensemble

We quantify uncertainty using an evidential framework based on Belief Function Theory (BFT). By employing MC-dropout, we generate an ensemble of density maps for a single input image. Rather than simple averaging, we model each pixel's output as a belief in the presence or absence of a target. A pixel-wise discounting mechanism reduces the influence of outliers within the ensemble. The final output includes a density estimate and an "Ignorance" map $\mathcal{M}(\Theta)$, which explicitly represents areas where the model lacks sufficient information.

Visual validation of the proposed framework: (a) original image patch, (b) predicted density map, and (c) ignorance map highlighting regions of high uncertainty.

Multi-Scale Evaluation Metrics

We introduce two key metrics to evaluate the reliability of uncertainty bounds:

  1. Prediction Error Probability (PEP): The frequency with which the ground truth falls outside the predicted uncertainty interval.
  2. Relative Imprecision (RI): The width of the uncertainty interval relative to the ground truth count.

By adjusting a scaling parameter, users can navigate the trade-off between imprecision and error probability, tailoring the model's behavior to specific safety requirements.

Results and Discussion

The proposed FE+LFE network demonstrates superior performance compared to standard architectures like U-Net, particularly in preserving spatial details for small targets. Visual analysis of the ignorance map reveals that uncertainty is highest at the boundaries of individual heads and in areas of high clutter, providing valuable diagnostic information. For a test patch with a ground truth of 12.3 people, the model predicted 12.01 with an uncertainty bound of $\pm 13\%$.

Comparison of the RI vs. PEP trade-off for (a) the proposed FE+LFE model and (b) the U-Net architecture across different scales ($S_1$ to $S_{10}$).

Quantitative results show that our framework effectively characterizes model reliability across multiple scales. Even when trained on a small subset of the Makkah Hajj dataset (35 images), the FE+LFE model maintains robust performance, matching or exceeding traditional SVM-based approaches.

Conclusion

By integrating Belief Function Theory with deep learning ensembles, we provide a robust framework for evaluating crowd density estimators. This approach moves beyond simple point estimates, offering actionable uncertainty bounds that are essential for safety-critical crowd monitoring applications. Future work will explore the generalization of this framework to other state-of-the-art architectures and larger, more diverse datasets.

Summary updated

Abstract

In this work, we use the Belief Function Theory which extends the probabilistic framework in order to provide uncertainty bounds to different categories of crowd density estimators. Our method allows us to compare the multi-scale performance of the estimators, and also to characterize their reliability for crowd monitoring applications requiring varying degrees of prudence.
BibTeX
@inproceedings{vandoni2019evaluating,
  author = {Jennifer Vandoni and Emanuel Aldea and Sylvie Le Hégarat-Mascle},
  title = {{Evaluating Crowd Density Estimators via Their Uncertainty Bounds}},
  booktitle = {ICIP},
  year = {2019},
  doi = {10.1109/icip.2019.8803522},
}
Figure