Learn to Scale: Generating Multipolar Normalized Density Map for Crowd Counting

Chenfeng Xu, Kai Qiu, Jianlong Fu, Song Bai, Yongchao Xu, Xiang Bai

TL;DR

This paper introduces a "Learning to Scale" (L2SM) module that mimics human behavior by automatically zooming in on dense crowd regions to improve counting accuracy. By using a novel multipolar center loss, the model normalizes diverse crowd densities into a few manageable clusters, allowing a single network to handle extreme scale variations that typically baffle standard convolutional neural networks.

What Problem Does This Paper Address?

Crowd counting—estimating the number of people in an image—is notoriously difficult because of extreme density variance. In a single dataset, you might see a sparse park scene alongside a sea of thousands of people at a protest.

Standard Convolutional Neural Networks (CNNs) have fixed receptive fields, meaning they look at the image through "windows" of a set size. When the density changes drastically, the visual patterns change (a phenomenon called "density pattern shift"). As shown in Figure 1, as density variance increases across datasets, the error rates of existing models skyrocket.

Figure 1. (a) Three examples from ShanghaiTech Part A dataset, which show extreme density variances. (b) Comparison of Mean Relative Error on four crowd counting datasets (the scale variances get larger from left to right) of different approaches. Results show the robustness of the proposed approach to extreme scale vari- ances. [best viewed in color].

Methods

The authors propose a two-stage pipeline to handle these shifts:

  1. Scale Preserving Network (SPN): This acts as the base model. It uses a VGG16 backbone and a feature pyramid to generate an initial, rough density map. This map helps identify which parts of the image are "dense" and need closer inspection.
  2. Learning to Scale Module (L2SM): This is the core innovation. The image is divided into $K \times K$ patches. For patches identified as dense, the module learns a specific scaling factor. It then "zooms in" on these patches (via bilinear upsampling) before re-predicting their density.
Figure 3. The overall pipeline showing the SPN generating an initial map and the L2SM module refining dense regions.

To make this work, they introduce Multipolar Center Loss (MPCL). Instead of letting the model scale patches arbitrarily, MPCL forces the relative density of patches to gravitate toward one of several learnable "centers" (density levels). This effectively "normalizes" the crowd patterns, making the dense regions look more like the sparse regions the network is already good at counting.

Datasets

The model was evaluated on four major benchmarks, ranging from moderate to extreme density:

  • ShanghaiTech (Part A & B): Standard benchmarks with varied urban scenes.
  • UCF_CC_50: A small but extremely challenging dataset of very dense crowds.
  • UCF-QNRF: A large-scale dataset featuring huge variations in both resolution and crowd density.

Key Insights and Ideas

  • Mimicking Human Intuition: The authors argue that when humans count a dense crowd, we naturally zoom in on congested areas to distinguish individuals. L2SM formalizes this "zoom-in" behavior for deep learning.
  • Density Normalization via Clustering: The "multipolar" aspect is clever. By treating density levels as clusters, the model doesn't have to learn an infinite range of scales. It just needs to learn how to transform a patch so it fits into one of a few "standard" density patterns.
  • Smart Ground-Truth (GT) Scaling: Simply stretching a density map during training doesn't work well. The authors found that they needed to re-compute the GT for scaled patches by increasing the distance between "blobs" (individual heads) while keeping the peak values the same. This directly combats the density pattern shift.
Figure 4. An example of ground-truth transformation for density map re-computation by enlarging the distance between blobs while keeping the original peaks, alleviating the density pattern shift be- tween sparse and dense regions.

Key Results

The proposed method achieved new state-of-the-art (SOTA) results across the board, with the most dramatic improvements seen in the densest datasets:

  • UCF_CC_50: Reduced Mean Absolute Error (MAE) from 258.4 to 188.4 (a 27.1% improvement).
  • UCF-QNRF: Reduced MAE from 132 to 104.7 (a 20.1% improvement).
  • ShanghaiTech Part A: Improved MAE by 4.2% over previous SOTA.

Qualitatively, Figure 6 shows how the L2SM refinement (middle columns) brings the predicted count much closer to the Ground Truth (GT) compared to the base SPN model.

Figure 6. Qualitative visualization of predicted density map on two examples. From left to right: original image, prediction given by SPN, re-predicted density map with L2SM on selected regions (englobed by black boxes), and ground-truth density map.

Limitations and Future Work

  • Computational Overhead: While the authors note the extra inference time is "negligible" (about 0.068s per image), the two-stage process is inherently slower than a single-pass network.
  • Patch Selection Dependency: The refinement stage depends on the initial SPN prediction. If the SPN completely misses a dense region, the L2SM won't have the chance to correct it.
  • Hyperparameter Sensitivity: The performance depends on choosing the right number of density groups ($G$) and centers ($C$), which might vary between different types of crowd environments.

Future work could involve making the patch selection and scaling fully differentiable or exploring how to integrate this "zoom" mechanism more tightly into the backbone architecture itself.

Summary updated

Abstract

Dense crowd counting aims to predict thousands of human instances from an image, by calculating integrals of a density map over image pixels. Existing approaches mainly suffer from the extreme density variances. Such density pattern shift poses challenges even for multi-scale model ensembling. In this paper, we propose a simple yet effective approach to tackle this problem. First, a patch-level density map is extracted by a density estimation model and further grouped into several density levels which are determined over full datasets. Second, each patch density map is automatically normalized by an online center learning strategy with a multipolar center loss. Such a design can significantly condense the density distribution into several clusters, and enable that the density variance can be learned by a single model. Extensive experiments demonstrate the superiority of the proposed method. Our work outperforms the state-of-the-art by 4.2%, 14.3%, 27.1% and 20.1% in MAE, on ShanghaiTech Part A, ShanghaiTech Part B, UCF_CC_50 and UCF-QNRF datasets, respectively.
BibTeX
@inproceedings{xu2019learn,
  author = {Chenfeng Xu and Kai Qiu and Jianlong Fu and Song Bai and Yongchao Xu and Xiang Bai},
  title = {{Learn to Scale: Generating Multipolar Normalized Density Map for Crowd Counting}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00847},
}
Figure