Abstract
Crowd counting based on density maps is generally regarded as a regression task. Deep learning is used to learn the mapping between image content and crowd density distribution. Although great success has been achieved, some pedestrians far away from the camera are difficult to be detected. And the number of hard examples is often larger. Existing methods with simple Euclidean distance algorithm indiscriminately optimize the hard and easy examples so that the densities of hard examples are usually incorrectly predicted to be lower or even zero, which results in large counting errors. To address this problem, we are the first to propose the hard example focusing (HEF) algorithm for the regression task of crowd counting. The HEF algorithm makes our model rapidly focus on hard examples by attenuating the contribution of easy examples. Then higher importance will be given to the hard examples with wrong estimations. Moreover, the scale variations in crowd scenes are large, and the scale annotations are labor-intensive and expensive. By proposing a multiscale semantic refining strategy, lower layers of our model can break through the limitation of deep learning to capture semantic features of different scales to sufficiently deal with the scale variation. We perform extensive experiments on six benchmark datasets to verify the provided method. Results indicate the superiority of our proposed method over the state-of-the-art methods. Moreover, our designed model is smaller and faster.
Introduction
Crowd counting via computer vision usually involves generating a "density map" where the sum of all pixels equals the total number of people in the image. While deep learning has made great strides, two major hurdles remain: the "Hard Sample Problem" and extreme "Scale Variation."
Pedestrians far from the camera appear as tiny, low-contrast clusters of pixels. Standard training typically uses a Euclidean (L2) loss, which treats every pixel with equal importance. Because "easy" samples (clearer people or background) dominate the loss calculation, models often fail to learn "hard" samples, incorrectly predicting zero or near-zero density for distant crowds. Additionally, a single image might contain a person right in front of the lens and a thousand people a block away. Capturing these features simultaneously usually requires expensive, manual "scale annotations" or massive, computationally heavy multi-column networks.
Methodology
The proposed SSR-HEF architecture introduces three key innovations to address these challenges:
Hard Example Focusing (HEF)
Inspired by the "Focal Loss" used in object detection, the authors adapted this concept for regression. They added a modulating factor to the standard L2 loss. If the model’s prediction for a pixel is far from the ground truth (a "hard" example), the weight remains high. If the prediction is already accurate (an "easy" example), the weight is attenuated. This forces the model to stop obsessing over easy pixels and "focus" on the difficult, distant pedestrians.
Multi-Scale Semantic Refining Module (SSRM)
To handle scale without manual labeling, the authors developed an automatic method to generate multi-scale segmentation maps. These maps are fed back into the network’s lower layers, helping the model learn semantic "crowd-ness" at different resolutions (1/2, 1/4, and 1/8 of the original size). This proves that lower layers can be trained to recognize complex multi-scale concepts, making the whole network more efficient.
Auxiliary Classification Task
The model includes a small branch that predicts a rough population class (e.g., "this image has between 50 and 100 people"). This acts as a global "sanity check" that helps the backbone network learn better general features.
Experimental Results
The researchers evaluated SSR-HEF on six major benchmark datasets: ShanghaiTech (Part A & B), UCF_CC_50, UCF-QNRF, WorldExpo’10, and JHU-CROWD.
Performance and Efficiency
SSR-HEF outperformed existing state-of-the-art (SOTA) methods across nearly all benchmarks. On the challenging UCF-QNRF dataset, it achieved a Mean Absolute Error (MAE) of 70.2, beating previous models like S-DCNet (104.4) and AMRNet (86.6).
The model is incredibly lightweight at just 2MB (2M parameters). In terms of inference speed, it clocked in at 0.003 seconds per image on an NVIDIA 1080Ti—roughly 3x faster than TopoCount, the previous speed leader.
Qualitative Analysis
Visualization of the density maps shows that SSR-HEF successfully recovers crowd clusters in the far background that other models completely ignore.
Conclusion
SSR-HEF demonstrates that the "focal" concept—usually reserved for discrete classification—is highly effective for continuous density regression. By down-weighting the "easy" background and clear foreground pixels, the model finally "sees" the tiny dots in the distance. While the model automates scale labeling, it still requires point annotations. Future work may explore self-supervised learning to further break the dependency on manual labeling.
Summary updated