Crowd Counting Via Perspective-Guided Fractional-Dilation Convolution

Zhaoyi Yan, Ruimao Zhang, Hongzhi Zhang, Qingfu Zhang, Wangmeng Zuo

TL;DR

Researchers have developed PFDNet, a novel neural network that handles the extreme scale variations in crowd images by using "fractional-dilation" convolutions. By using perspective information to continuously adjust the receptive field at every pixel, the model achieves state-of-the-art accuracy while remaining more computationally efficient than traditional multi-column architectures.

What Problem Does This Paper Address?

Crowd counting is notoriously difficult because of the perspective effect: pedestrians close to the camera appear much larger than those in the distance.

To handle these scale variations, previous researchers typically used "multi-column" networks (different branches for different scales) or "dilated convolutions" (which expand the kernel's reach). However, multi-column networks are computationally heavy and often learn redundant features. Standard dilated convolutions are also limited because they only support discrete, integer-based expansion rates (e.g., a dilation of 1, 2, or 3). This creates a "jumpy" adaptation to what is actually a smooth, continuous change in scale across an image.

Methods

The authors introduce the Perspective-guided Fractional-Dilation Network (PFDNet). Its core components are:

  1. Perspective-guided Fractional-Dilation Convolution (PFC): Unlike standard convolutions that sample from a fixed grid, PFC allows for fractional dilation rates. If the perspective map suggests a pedestrian is at a scale of 1.5x, the convolution samples at that exact fractional interval. Since these points don't land perfectly on the pixel grid, the model uses bilinear interpolation to estimate the feature values.
  2. Continuous Scale Modeling: By mapping perspective values to dilation rates via a learnable sigmoid-like function, the network can adapt its receptive field smoothly across the entire spatial domain.
  3. PENet (Perspective Estimation Network): Perspective maps aren't always available in datasets. The authors designed a separate branch using an auto-encoder architecture to estimate these maps. They use a clever three-phase training strategy (pre-training on known maps, then images, then joint fine-tuning) to ensure the estimated perspective is robust even in a weakly-supervised setting.

Datasets

The model was evaluated on several major crowd counting and object counting benchmarks:

  • ShanghaiTech (Parts A and B): Standard benchmarks for diverse crowd densities.
  • WorldExpo’10: Focuses on cross-scene counting.
  • UCF-QNRF & UCF_CC_50: Extremely dense crowds with high-resolution images.
  • TRANCOS: A public traffic dataset used to test the model's generalization to vehicle counting.

Key Insights and Ideas

  • Fractional > Discrete: Moving from integer dilation to fractional dilation allows the network to precisely match the physical scale of objects, rather than approximating them with the nearest available fixed filter size.
  • Avoiding "Feature Confusion": The authors point out that previous methods using Gaussian smoothing to handle scale (like PGCNet) suffer from overlapping sampling regions, which blurs and confuses features. Because PFC uses point-based bilinear interpolation, it maintains much cleaner feature representation.
  • Efficiency Matters: By using a single-column backbone (VGG16-BN) and replacing standard dilated layers with PFC modules, the authors achieved better results than complex multi-branch networks with significantly less computational overhead (approx. 70ms for a 1024x768 image).

Key Results

PFDNet consistently outperformed or matched state-of-the-art methods across all tested datasets:

  • ShanghaiTech A: Achieved a Mean Absolute Error (MAE) of 53.8, a significant improvement over the baseline CSRNet (68.2).
  • UCF-QNRF: Reached an MAE of 84.3, ranking it among the top performers for large-scale dense crowds.
  • Generalization: On the TRANCOS vehicle dataset, it achieved an MAE of 3.06, proving the architecture isn't just for "heads" but for any perspective-driven counting task.
  • Speed: The PFC module is roughly 3x faster than previous perspective-aware convolutions (like PGCNet) because it avoids expensive matrix multiplications for Gaussian kernels.

Limitations and Future Work

While PFDNet is highly effective, the authors note a few areas for improvement:

  • Optimal Placement: The current architecture replaces all six dilated convolutions in the backbone with PFCs. The authors suggest that the "optimal" number and placement of these layers across different backbones (like ResNet) could be further explored.
  • Perspective Dependency: Although PENet estimates perspective maps well, the system still fundamentally relies on the assumption that scale changes are primarily driven by a linear perspective ground plane. Scenes with complex 3D topography (e.g., stairs or multi-level balconies) might challenge this assumption.

Future work could involve applying this fractional-dilation concept to other computer vision tasks where scale is a factor, such as object detection or semantic segmentation.

Summary updated

Abstract

Crowd counting is critical for numerous video surveillance scenarios. One of the main issues in this task is how to handle the dramatic scale variations of pedestrians caused by the perspective effect. To address this issue, this paper proposes a novel convolution neural network-based crowd counting method, termed Perspective-guided Fractional-Dilation Network (PFDNet). By modeling the continuous scale variations, the proposed PFDNet is able to select the proper fractional-dilation kernels for adapting to different spatial locations. It significantly improves the flexibility of the state-of-the-arts that only consider the discrete representative scales. In addition, by avoiding the multi-scale or multi-column architecture that used in other methods, it is computationally more efficient. In practice, the proposed PFDNet is constructed by stacking multiple Perspective-guided Fractional-Dilation Convolutions (PFC) on a VGG16-BN backbone. By introducing a novel generalized dilation convolution operation, the PFC can handle fractional dilation ratios in the spatial domain under the guidance of perspective annotations, achieving continuous scales modeling of pedestrians. To deal with the problem of unavailable perspective information in some cases, we further introduce an effective perspective estimation branch to the proposed PFDNet, which can be trained in either supervised or weakly-supervised setting once the branch has been pre-trained. Extensive experiments show that the proposed PFDNet outperforms state-of-the-art methods on ShanghaiTech A, ShanghaiTech B, WorldExpo’10, UCF-QNRF, UCF_CC_50 and TRANCOS dataset, achieving MAE 53.8, 6.5, 6.8, 84.3205.8, and 3.06 respectively.
BibTeX
@article{yan2022crowd,
  author = {Zhaoyi Yan and Ruimao Zhang and Hongzhi Zhang and Qingfu Zhang and Wangmeng Zuo},
  title = {{Crowd Counting Via Perspective-Guided Fractional-Dilation Convolution}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=HzWBiy17qv6},
}

Loading related content...

Figure