Abstract
Crowd counting based on density maps is generally regarded as a regression task. Deep learning is used to learn the mapping between image content and crowd density distribution. Although great success has been achieved, some pedestrians far away from the camera are difficult to be detected, and the number of these "hard examples" is often larger. Existing methods using simple Euclidean distance algorithms indiscriminately optimize hard and easy examples, often resulting in incorrect zero or low-density predictions for distant crowds. To address this, we propose the Hard Example Focusing (HEF) algorithm for crowd counting regression. HEF forces the model to focus on hard examples by attenuating the contribution of easy examples. Additionally, we introduce a multi-Scale Semantic Refining (SSR) strategy to capture semantic features across different scales early in the network. Extensive experiments on six benchmark datasets demonstrate that our method outperforms state-of-the-art models while being significantly smaller and faster.
Introduction
Crowd counting via density map estimation is a critical task for public safety and urban management. However, two major bottlenecks persist: extreme scale variation and the "hard example" problem. In typical scenes, pedestrians near the camera appear large and clear, while those in the distance are tiny and dense. Standard models using Euclidean (L2) loss are often dominated by "easy" examples—background pixels and large, clear pedestrians—causing them to ignore the tiny, distant clusters that constitute the most challenging part of the count. This results in significant undercounting in dense, distant regions.
Methodology
The SSR-HEF framework introduces two primary modules to address these challenges: the Multi-Scale Semantic Refining Module (SSRM) and the Hard Example Focusing Module (HEFM).
Multi-Scale Semantic Refining Module (SSRM)
To handle scale variation without expensive manual scale annotations, SSRM utilizes an automatic labeling method to generate multi-scale segmentation maps. These maps act as a "semantic prior," informing the network about crowd locations at various resolutions (1/2, 1/4, and 1/8 scales). By feeding these priors back into the lower layers of the network, the model learns to adaptively refine features for different scales early in the pipeline, rather than relying solely on deep layers for semantic understanding.
Hard Example Focusing Module (HEFM)
The HEFM introduces the Hard Example Focusing (HEF) algorithm, which adapts the concept of Focal Loss from classification to the regression task of crowd counting. By introducing a modulating factor to the standard Euclidean distance, the loss function is attenuated for pixels where the model is already confident and accurate ("easy" examples). This forces the training gradient to focus almost entirely on correcting the difficult, distant pedestrians that are typically ignored.
As illustrated in the comparison above, standard models (c) often fail to detect distant crowds in dense clusters (red box), leading to a significant undercount (224 vs. 357). SSR-HEF (d) successfully recovers these hard examples, bringing the estimated count much closer to the ground truth.
Experimental Results
SSR-HEF was evaluated on six major benchmarks, including ShanghaiTech, UCF-QNRF, and JHU-CROWD, demonstrating superior performance across several metrics:
- Accuracy: On the ShanghaiTech Part A dataset, SSR-HEF achieved a Mean Absolute Error (MAE) of 55.0, outperforming significantly larger models like S-DCNet (58.3).
- Efficiency: The model is remarkably lightweight, requiring only 2 million parameters—making it up to 50x smaller than many state-of-the-art competitors.
- Speed: It achieves real-time performance, processing images in approximately 0.003 seconds (333 FPS) on an NVIDIA 1080Ti, roughly 3x faster than previous high-speed models.
Conclusion
SSR-HEF solves the dual challenges of scale variation and hard example neglect through a lightweight and efficient architecture. By prioritizing difficult, distant pedestrians and refining multi-scale semantics early in the network, it achieves state-of-the-art results with minimal computational overhead. Future work will focus on self-supervised learning to reduce the reliance on manual "dot" annotations for large-scale datasets.
Summary updated