Crowd Counting via Perspective-Guided Fractional-Dilation Convolution

Zhaoyi Yan, Ruimao Zhang, Hongzhi Zhang, Qingfu Zhang, Wangmeng Zuo

Abstract

Crowd counting is critical for numerous video surveillance scenarios. One of the main issues in this task is how to handle the dramatic scale variations of pedestrians caused by the perspective effect. To address this issue, this paper proposes a novel convolution neural network-based crowd counting method, termed Perspective-guided Fractional-Dilation Network (PFDNet). By modeling the continuous scale variations, the proposed PFDNet is able to select the proper fractional dilation kernels for adapting to different spatial locations. It significantly improves the flexibility of the state-of-the-arts that only consider the discrete representative scales. In addition, by avoiding the multi-scale or multi-column architecture that used in other methods, it is computationally more efficient. In practice, the proposed PFDNet is constructed by stacking multiple Perspective-guided Fractional-Dilation Convolutions (PFC) on a VGG16-BN backbone. By introducing a novel generalized dilation convolution operation, the PFC can handle fractional dilation ratios in the spatial domain under the guidance of perspective annotations, achieving continuous scales modeling of pedestrians. To deal with the problem of unavailable perspective information in some cases, we further introduce an effective perspective estimation branch to the proposed PFDNet, which can be trained in either supervised or weakly-supervised setting once the branch has been pre-trained. Extensive experiments show that the proposed PFDNet outperforms state-of-the-art methods on ShanghaiTech A, ShanghaiTech B, WorldExpo'10, UCF-QNRF, UCF_CC_50 and TRANCOS dataset, achieving MAE 53.8, 6.5, 6.8, 84.3, 205.8, and 3.06 respectively.

Introduction

Crowd counting in real-world scenarios is notoriously difficult because of the perspective effect: pedestrians close to the camera appear much larger than those in the distance. To handle these scale variations, previous research typically utilized "multi-column" networks (different branches for different scales) or "dilated convolutions" (which expand the kernel's reach). However, multi-column networks are computationally heavy and often learn redundant features. Standard dilated convolutions are also limited because they only support discrete, integer-based expansion rates (e.g., a dilation of 1, 2, or 3). This creates a "jumpy" adaptation to what is actually a smooth, continuous change in scale across an image.

Methodology

The authors introduce the Perspective-guided Fractional-Dilation Network (PFDNet) to address these limitations through two primary innovations:

Perspective-guided Fractional-Dilation Convolution (PFC)

Unlike standard convolutions that sample from a fixed grid, PFC allows for fractional dilation rates. If the perspective map suggests a pedestrian is at a scale of 1.5x, the convolution samples at that exact fractional interval. Since these points do not land perfectly on the pixel grid, the model uses bilinear interpolation to estimate the feature values. By mapping perspective values to dilation rates via a learnable sigmoid-like function, the network can adapt its receptive field smoothly across the entire spatial domain.

Perspective Estimation Network (PENet)

Since perspective maps are not always available in datasets, the authors designed a separate branch using an auto-encoder architecture to estimate these maps. They use a three-phase training strategy—pre-training on known maps, then images, then joint fine-tuning—to ensure the estimated perspective is robust even in a weakly-supervised setting.

Experimental Results

PFDNet consistently outperformed or matched state-of-the-art methods across all tested datasets. Notably, on ShanghaiTech A, it achieved a Mean Absolute Error (MAE) of 53.8, a significant improvement over the baseline CSRNet (68.2). The model also reached an MAE of 84.3 on UCF-QNRF, ranking it among the top performers for large-scale dense crowds.

Comparison of crowd counting results showing (a) Input image, (b) CSRNet baseline, (c) Proposed PFDNet, and (d) Ground-truth density maps.

Beyond accuracy, efficiency is a key advantage. By using a single-column backbone (VGG16-BN) and replacing standard dilated layers with PFC modules, the authors achieved better results than complex multi-branch networks with significantly less computational overhead (approx. 70ms for a 1024x768 image). The PFC module is roughly 3x faster than previous perspective-aware convolutions because it avoids expensive matrix multiplications for Gaussian kernels.

Conclusion

PFDNet demonstrates that moving from integer dilation to fractional dilation allows networks to precisely match the physical scale of objects. While highly effective, the authors note that the system fundamentally relies on the assumption that scale changes are driven by a linear perspective ground plane. Future work could involve applying this fractional-dilation concept to other computer vision tasks where scale is a factor, such as object detection or semantic segmentation.

Summary updated

Abstract

Crowd counting is critical for numerous video surveillance scenarios. One of the main issues in this task is how to handle the dramatic scale variations of pedestrians caused by the perspective effect. To address this issue, this paper proposes a novel convolution neural network-based crowd counting method, termed Perspective-guided Fractional-Dilation Network (PFDNet). By modeling the continuous scale variations, the proposed PFDNet is able to select the proper fractional dilation kernels for adapting to different spatial locations. It significantly improves the flexibility of the state-of-the-arts that only consider the discrete representative scales. In addition, by avoiding the multi-scale or multi-column architecture that used in other methods, it is computationally more efficient. In practice, the proposed PFDNet is constructed by stacking multiple Perspective-guided Fractional-Dilation Convolutions (PFC) on a VGG16-BN backbone. By introducing a novel generalized dilation convolution operation, the PFC can handle fractional dilation ratios in the spatial domain under the guidance of perspective annotations, achieving continuous scales modeling of pedestrians. To deal with the problem of unavailable perspective information in some cases, we further introduce an effective perspective estimation branch to the proposed PFDNet, which can be trained in either supervised or weakly-supervised setting once the branch has been pre-trained. Extensive experiments show that the proposed PFDNet outperforms state-of-the-art methods on ShanghaiTech A, ShanghaiTech B, WorldExpo'10, UCF-QNRF, UCF_CC_50 and TRANCOS dataset, achieving MAE 53.8, 6.5, 6.8, 84.3, 205.8, and 3.06 respectively.
BibTeX
@article{yan2022crowd,
  author = {Zhaoyi Yan and Ruimao Zhang and Hongzhi Zhang and Qingfu Zhang and Wangmeng Zuo},
  title = {{Crowd Counting via Perspective-Guided Fractional-Dilation Convolution}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=HzWBiy17qv6},
}
Figure