Crowd Counting via Weighted VLAD on Dense Attribute Feature Maps

Biyun Sheng, Chunhua Shen, Guosheng Lin, Jun Li, Wankou Yang, Changyin Sun

Introduction

Crowd counting—the estimation of the number of people in a scene—is a fundamental task in video surveillance and urban management. While regression-based models have seen significant improvements, they often struggle with discriminative representation, particularly in distinguishing human clusters from complex background textures. Traditional features like edge orientation or texture patterns often lack semantic "understanding," leading to errors where background elements are mistaken for crowds. This paper addresses these challenges by proposing a framework that leverages pixel-level semantic attributes and localized spatial information.

Methodology

The proposed framework introduces a three-stage pipeline to enhance the discriminative power of crowd-counting features.

Dense Attribute Feature Mapping

The first stage transforms raw image pixels into dense attribute maps. By employing a Convolutional Neural Network (CNN) trained on the Cityscapes dataset for semantic segmentation, the model maps each pixel to a probability vector representing various semantic classes (e.g., person, road, tree). This captures the co-occurrence of patterns—for instance, recognizing that a "person" is more likely to be found on a "road" than in a "tree."

Dense attribute feature mapping: input frames (top) and their corresponding semantic segmentation maps (bottom).

Locality-Aware Features (LAF)

To preserve spatial information often lost in global features, the image is partitioned into cells. Within each cell, a spatial pyramid (an $M$-grid structure) is applied to capture local context. This approach allows the model to understand density variations within small regions, which is essential for counting in non-uniform crowds.

Weighted VLAD (W-VLAD)

The final stage involves encoding the extracted features using a novel Weighted Vector of Locally Aggregated Descriptors (W-VLAD) method. Unlike standard VLAD, which uses hard assignment, W-VLAD employs soft assignment based on the distance to the $\kappa$ nearest neighbors in the feature space. This results in a more nuanced and robust image representation, reducing reconstruction errors and improving regression accuracy.

Experimental Results

The researchers evaluated the proposed method on three benchmark datasets: the Mall dataset, the UCSD dataset, and the Caltech 10X dataset.

  • Mall Dataset: The method achieved a Mean Absolute Error (MAE) of 2.86 and a Mean Squared Error (MSE) of 13.05, outperforming previous state-of-the-art models such as Multi-Output Ridge Regression (MORR).
  • UCSD Dataset: The framework yielded an MAE of 2.41, demonstrating competitive performance in scenes with smaller pedestrians and specific region-of-interest constraints.
  • Caltech 10X: Even in challenging low-visibility scenarios with small or occluded individuals, the method remained robust, achieving an MAE of 1.36.

Ablation studies confirmed that the combination of Locality-Aware Features and W-VLAD consistently outperformed baselines using only holistic features or standard VLAD encoding.

Discussion and Conclusion

By integrating semantic attribute maps, the proposed model gains a "common sense" filter that effectively distinguishes people from background clutter. The combination of semantic context, local spatial cues, and soft-assignment encoding provides a superior representation for crowd counting. While the multi-stage process is computationally more intensive than traditional methods and can be sensitive to extremely dark environments, it represents a significant step forward in robust crowd estimation. Future research will focus on optimizing the pipeline for real-time surveillance applications and improving performance under dynamic background conditions.

Summary updated

Abstract

Crowd counting is an important task in computer vision, which has many applications in video surveillance. Although the regression-based framework has achieved great improvements for crowd counting, how to improve the discriminative power of image representation is still an open problem. Conventional holistic features used in crowd counting often fail to capture semantic attributes and spatial cues of the image. In this paper, we propose integrating semantic information into learning locality-aware feature sets for accurate crowd counting. First, with the help of convolutional neural network (CNN), the original pixel space is mapped onto a dense attribute feature map, where each dimension of the pixel-wise feature indicates the probabilistic strength of a certain semantic class. Then, locality-aware features (LAF) built on the idea of spatial pyramids on neighboring patches are proposed to explore more spatial context and local information. Finally, the traditional VLAD encoding method is extended to a more generalized form in which diverse coefficient weights are taken into consideration. Experimental results validate the effectiveness of our presented method.
BibTeX
@article{sheng2018crowd,
  author = {Biyun Sheng and Chunhua Shen and Guosheng Lin and Jun Li and Wankou Yang and Changyin Sun},
  title = {{Crowd Counting via Weighted VLAD on Dense Attribute Feature Maps}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2018},
  doi = {10.1109/tcsvt.2016.2637379},
}
Figure