Offset-decoupled deformable convolution for efficient crowd counting

Xin Zhong, Jing Qin, Mingyue Guo, Wangmeng Zuo, Weigang LuScientific 2022 · DOI

TL;DR

This paper introduces Offset-Decoupled Deformable Convolution (ODConv), a novel module for crowd counting that addresses the issue of randomly initialized offsets in deformable convolutions. By decoupling the offset learning into an initial offset and a scale map, ODConv achieves more stable and effective feature extraction, leading to state-of-the-art performance on several benchmark datasets with minimal computational overhead.

What Problem Does This Paper Address?

Accurately counting people in crowded scenes is a long-standing challenge in computer vision, primarily due to significant variations in object scale and density. While Convolutional Neural Networks (CNNs) have shown great promise, their fixed geometric structures struggle to adapt to these scale variations. Deformable convolutions offer a solution by allowing the network to learn adaptive sampling locations, but their effectiveness is hampered by randomly initialized offsets. This randomness can lead to disordered sampling points, weakening feature extraction and limiting the potential of deformable convolutions. The paper aims to solve this by proposing a method to better constrain and learn these offsets for improved crowd counting accuracy.

Methods

The core contribution is the Offset-Decoupled Deformable Convolution (ODConv). Unlike standard deformable convolutions where offsets are directly learned from input features, ODConv decouples this process. It splits the offset generation into two components:

  • Pre-offset Map: Learned through a "pre-offset convolution," this map is constrained by a loss function to be close to zero after a tanh activation, effectively acting as an initial, stable offset.
  • Scale Map: Learned through a "scale convolution" with a ReLU activation to ensure non-negativity, this map controls the magnitude of the initial offsets.

These two maps are then multiplied to produce the final offset map used by the deformable convolution. This decoupling allows for more controlled and stable learning of offsets, preventing the disorderliness seen in traditional deformable convolutions. The authors integrate ODConv into the CSRNet backbone, replacing its last dilated convolution layer.

Datasets

The proposed ODConv was evaluated on four widely used crowd counting datasets:

  • ShanghaiTech A: Contains images from streets and campuses with varying crowd densities.
  • ShanghaiTech B: Features high-resolution images of extremely dense crowds.
  • UCF-QNRF: Known for its high resolution and high crowd density, with a large number of images for testing.
  • UCF_CC_50: A dataset with varied crowd densities, used with five-fold cross-validation.

Key Insights and Ideas

  • The "randomness problem" of deformable convolution offsets: The paper clearly articulates how random initialization of offsets in deformable convolutions leads to disordered sampling points and hinders effective feature extraction.
  • Decoupling for stability: The core idea of separating the offset learning into an initial offset and a scale factor is a clever way to introduce stability and control into the deformable convolution process.
  • Constrained learning: The introduction of specific loss functions for the pre-offset and scale maps is crucial for guiding the learning process towards optimal values.
  • Minimal computational overhead: ODConv achieves improved performance without significantly increasing computational complexity, making it a practical solution.
  • Last layer replacement is sufficient: Experiments suggest that replacing only the last dilated convolution layer with ODConv is an effective strategy, simplifying integration into existing architectures.

Key Results

The ODConv method demonstrated significant improvements across all tested datasets:

  • ShanghaiTech A: Achieved an MAE of 62.3, outperforming state-of-the-art methods.
  • ShanghaiTech B: Achieved an MAE of 8.3, a 0.7 MAE improvement over the baseline.
  • UCF-QNRF: Achieved an MAE of 91.9, a gain of 3.6 MAE over the baseline CSRNet.
  • UCF_CC_50: Achieved an MAE of 159.3, a gain of 15.3 MAE over the baseline CSRNet.

Qualitatively, visualizations show that ODConv produces more evenly distributed sampling points compared to standard deformable convolution, leading to better feature aggregation.

Limitations and Future Work

  • Number of deformable layers: Ablation studies suggest that performance is better when fewer deformable layers are used, indicating that simply replacing the last layer is optimal. This implies that the benefits of ODConv are most pronounced when applied to the final feature extraction stage.
  • Extensibility: The paper demonstrates ODConv's effectiveness on different backbones (CSRNet and ResNet-50) through T-test analysis, showing statistically significant improvements. This suggests potential for broader application.
  • Future work: While not explicitly detailed, the success of ODConv on various datasets and backbones suggests future work could involve exploring its integration into more complex architectures or applying it to other computer vision tasks where scale variation is a challenge.

Summary updated

Abstract

Crowd counting is considered a challenging issue in computer vision. One of the most critical challenges in crowd counting is considering the impact of scale variations. Compared with other methods, better performance is achieved with CNN-based methods. However, given the limit of fixed geometric structures, the head-scale features are not completely obtained. Deformable convolution with additional offsets is widely used in the fields of image classification and pattern recognition, as it can successfully exploit the potential of spatial information. However, owing to the randomly generated parameters of offsets in network initialization, the sampling points of the deformable convolution are disorderly stacked, weakening the effectiveness of feature extraction. To handle the invalid learning of offsets and the inefficient utilization of deformable convolution, an offset-decoupled deformable convolution (ODConv) is proposed in this paper. It can completely obtain information within the effective region of sampling points, leading to better performance. In extensive experiments, average MAE of 62.3, 8.3, 91.9, and 159.3 are achieved using our method on the ShanghaiTech A, ShanghaiTech B, UCF-QNRF, and UCF_CC_50 datasets, respectively, outperforming the state-of-the-art methods and validating the effectiveness of the proposed ODConv.
BibTeX
@inproceedings{zhong2022offset,
  author = {Xin Zhong and Jing Qin and Mingyue Guo and Wangmeng Zuo and Weigang Lu},
  title = {{Offset-decoupled deformable convolution for efficient crowd counting}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-16415-9},
}
Figure