SSR-HEF: Crowd Counting with Multi-Scale Semantic Refining and Hard Example Focusing

Jiwei Chen, Kewei Wang, Wen Su, Zengfu Wang

Abstract

Crowd counting based on density maps is generally regarded as a regression task. Deep learning is used to learn the mapping between image content and crowd density distribution. Although great success has been achieved, some pedestrians far away from the camera are difficult to be detected, and the number of these "hard examples" is often larger. Existing methods using simple Euclidean distance algorithms indiscriminately optimize hard and easy examples, often resulting in incorrect zero or low-density predictions for distant crowds. To address this, we propose the Hard Example Focusing (HEF) algorithm for crowd counting regression. HEF forces the model to focus on hard examples by attenuating the contribution of easy examples. Additionally, we introduce a multi-Scale Semantic Refining (SSR) strategy to capture semantic features across different scales early in the network. Extensive experiments on six benchmark datasets demonstrate that our method outperforms state-of-the-art models while being significantly smaller and faster.

Introduction

Crowd counting via density map estimation is a critical task for public safety and urban management. However, two major bottlenecks persist: extreme scale variation and the "hard example" problem. In typical scenes, pedestrians near the camera appear large and clear, while those in the distance are tiny and dense. Standard models using Euclidean (L2) loss are often dominated by "easy" examples—background pixels and large, clear pedestrians—causing them to ignore the tiny, distant clusters that constitute the most challenging part of the count. This results in significant undercounting in dense, distant regions.

Methodology

The SSR-HEF framework introduces two primary modules to address these challenges: the Multi-Scale Semantic Refining Module (SSRM) and the Hard Example Focusing Module (HEFM).

Multi-Scale Semantic Refining Module (SSRM)

To handle scale variation without expensive manual scale annotations, SSRM utilizes an automatic labeling method to generate multi-scale segmentation maps. These maps act as a "semantic prior," informing the network about crowd locations at various resolutions (1/2, 1/4, and 1/8 scales). By feeding these priors back into the lower layers of the network, the model learns to adaptively refine features for different scales early in the pipeline, rather than relying solely on deep layers for semantic understanding.

Hard Example Focusing Module (HEFM)

The HEFM introduces the Hard Example Focusing (HEF) algorithm, which adapts the concept of Focal Loss from classification to the regression task of crowd counting. By introducing a modulating factor to the standard Euclidean distance, the loss function is attenuated for pixels where the model is already confident and accurate ("easy" examples). This forces the training gradient to focus almost entirely on correcting the difficult, distant pedestrians that are typically ignored.

Visual comparison of crowd counting performance: (a) input image, (b) ground truth, (c) baseline prediction, and (d) SSR-HEF prediction focusing on hard examples in the red box.

As illustrated in the comparison above, standard models (c) often fail to detect distant crowds in dense clusters (red box), leading to a significant undercount (224 vs. 357). SSR-HEF (d) successfully recovers these hard examples, bringing the estimated count much closer to the ground truth.

Experimental Results

SSR-HEF was evaluated on six major benchmarks, including ShanghaiTech, UCF-QNRF, and JHU-CROWD, demonstrating superior performance across several metrics:

  • Accuracy: On the ShanghaiTech Part A dataset, SSR-HEF achieved a Mean Absolute Error (MAE) of 55.0, outperforming significantly larger models like S-DCNet (58.3).
  • Efficiency: The model is remarkably lightweight, requiring only 2 million parameters—making it up to 50x smaller than many state-of-the-art competitors.
  • Speed: It achieves real-time performance, processing images in approximately 0.003 seconds (333 FPS) on an NVIDIA 1080Ti, roughly 3x faster than previous high-speed models.

Conclusion

SSR-HEF solves the dual challenges of scale variation and hard example neglect through a lightweight and efficient architecture. By prioritizing difficult, distant pedestrians and refining multi-scale semantics early in the network, it achieves state-of-the-art results with minimal computational overhead. Future work will focus on self-supervised learning to reduce the reliance on manual "dot" annotations for large-scale datasets.

Summary updated

Abstract

Crowd counting based on density maps is generally regarded as a regression task.Deep learning is used to learn the mapping between image content and crowd density distribution. Although great success has been achieved, some pedestrians far away from the camera are difficult to be detected. And the number of hard examples is often larger. Existing methods with simple Euclidean distance algorithm indiscriminately optimize the hard and easy examples so that the densities of hard examples are usually incorrectly predicted to be lower or even zero, which results in large counting errors. To address this problem, we are the first to propose the Hard Example Focusing(HEF) algorithm for the regression task of crowd counting. The HEF algorithm makes our model rapidly focus on hard examples by attenuating the contribution of easy examples.Then higher importance will be given to the hard examples with wrong estimations. Moreover, the scale variations in crowd scenes are large, and the scale annotations are labor-intensive and expensive. By proposing a multi-Scale Semantic Refining (SSR) strategy, lower layers of our model can break through the limitation of deep learning to capture semantic features of different scales to sufficiently deal with the scale variation. We perform extensive experiments on six benchmark datasets to verify the proposed method. Results indicate the superiority of our proposed method over the state-of-the-art methods. Moreover, our designed model is smaller and faster.
BibTeX
@article{chen2022ssr,
  author = {Jiwei Chen and Kewei Wang and Wen Su and Zengfu Wang},
  title = {{SSR-HEF: Crowd Counting with Multi-Scale Semantic Refining and Hard Example Focusing}},
  journal = {IEEE Trans. Industrial Informatics},
  year = {2022},
  url = {https://openreview.net/forum?id=NUK8FqwxbN},
}
Figure