HA-CCN: Hierarchical Attention-Based Crowd Counting Network

Vishwanath A. Sindagi, Vishal M. Patel

TL;DR

This paper introduces HA-CCN, a novel crowd counting network that uses hierarchical attention mechanisms to improve feature representation for dense and varied crowd scenes. It also proposes a weakly supervised learning approach that adapts existing models to new datasets using only image-level labels, significantly reducing annotation costs.

What Problem Does This Paper Address?

Accurately counting people in images, especially in highly congested or varied scenes, is a challenging computer vision problem. Existing methods often struggle with issues like perspective distortion, extreme scale variations, and occlusions. While deep learning has improved performance, many state-of-the-art methods are complex to train, require extensive point-wise annotations for new datasets, and don't generalize well across different environments. This paper aims to address these limitations by proposing a more effective network architecture and a more practical, weakly supervised adaptation strategy.

Methods

The core of the proposed method is the Hierarchical Attention-based Crowd Counting Network (HA-CCN), built upon the VGG16 architecture. HA-CCN incorporates two main attention modules:

  • Spatial Attention Module (SAM): This module operates on lower-level features (from the conv3 layer). It leverages foreground-background segmentation masks (derived from existing density maps) to infuse spatial awareness into the feature maps. This helps the network focus on relevant regions and suppress background noise.
  • Global Attention Modules (GAM): These modules are applied to higher-level features (from conv4 and conv5 layers). They enhance feature representation by selectively focusing on important channels and suppressing less relevant ones, effectively capturing channel-wise information.

The paper also introduces a novel weakly supervised learning setup for adapting the trained HA-CCN to new datasets. This approach uses image-level labels (e.g., "low density," "high density") to train a Class Activation Map (CAM) module. This CAM module then generates "pseudo ground-truth" density maps, which are used to fine-tune the counting network without requiring expensive point-wise annotations.

Datasets

The paper uses the following datasets for training and evaluation:

  • ShanghaiTech: A large-scale dataset with two parts (Part A and Part B), containing a total of 1198 images and over 330,000 annotations.
  • UCF_CROWD_50: A dataset with 50 annotated images of varying resolutions.
  • UCF-QNRF: A recent, large-scale dataset with 1,535 high-quality images and 1.25 million annotations.
  • WorldExpo '10: Used for cross-dataset evaluation.

Key Insights and Ideas

  • Hierarchical Attention is Key: Applying different types of attention (spatial and global) at different network levels is crucial for effectively handling the complexities of crowd scenes. Spatial attention helps with localization and background suppression, while global attention refines feature representation across channels.
  • Segmentation for Spatial Attention: Using segmentation masks (even noisy ones derived from density maps) to guide spatial attention is a clever way to inject semantic understanding into early network layers without requiring entirely new annotation schemes.
  • Weak Supervision for Generalization: The proposed weakly supervised approach using image-level labels is a significant step towards making crowd counting models more adaptable to new, unseen environments without the prohibitive cost of dense annotations. This is a novel contribution to the field.
  • Pseudo Ground-Truth Generation: The method of generating pseudo ground-truth density maps from image-level classifications is an effective way to bridge the gap between coarse labels and the fine-grained density maps needed for training.

Key Results

  • State-of-the-Art Performance: HA-CCN achieves state-of-the-art results on the ShanghaiTech, UCF_CROWD_50, and UCF-QNRF datasets, outperforming several recent methods in terms of Mean Absolute Error (MAE) and Mean Squared Error (MSE). For example, on ShanghaiTech Part B, HA-CCN achieved an MAE of 8.1, significantly lower than other methods.
  • Ablation Study Benefits: The ablation study demonstrates that each component of HA-CCN contributes to performance improvement. Specifically, adding SAM and GAM modules leads to substantial error reductions compared to a baseline VGG16 with multi-scale feature concatenation. Explicitly supervised SAM also performs better than self-supervised SAM.
  • Improved Cross-Dataset Performance: The proposed weakly supervised learning setup significantly improves generalization to new datasets. For instance, when adapting to the WorldExpo '10 dataset, HA-CCN showed a much smaller drop in performance compared to other methods.
  • Qualitative Improvements: Visualizations show that HA-CCN effectively suppresses background regions and produces more accurate density maps, especially in challenging, dense crowd scenarios.

Limitations and Future Work

  • Noisy Annotations: The paper acknowledges that the segmentation masks derived from existing density maps can be noisy, but demonstrates that the attention mechanism still yields considerable gains.
  • Generalization Gap: While the weakly supervised approach improves generalization, a performance gap still exists compared to fully supervised methods trained on the target dataset.
  • Future Work: The authors plan to explore better ways of incorporating features from different layers and extend the framework to other backbone networks. They also aim to investigate other forms of weakly supervised and semi-supervised learning to further enhance cross-dataset performance.

Summary updated

Abstract

Single image-based crowd counting has recently witnessed increased focus, but many leading methods are far from optimal, especially in highly congested scenes. In this paper, we present the Hierarchical Attention-based Crowd Counting Network (HA-CCN) that employs attention mechanisms at various levels to selectively enhance the features of the network. The proposed method, which is based on the VGG16 network, consists of a spatial attention module (SAM) and a set of global attention modules (GAM). SAM enhances low-level features in the network by infusing spatial segmentation information, whereas the GAM focuses on enhancing channel-wise information in the higher level layers. The proposed method is a single-step training framework, simple to implement and achieves the state-of-the-art results on different datasets. Furthermore, we extend the proposed counting network by introducing a novel set-up to adapt the network to different scenes and datasets via weak supervision using image-level labels. This new set up reduces the burden of acquiring labor intensive point-wise annotations for new datasets while improving the cross-dataset performance.
BibTeX
@article{sindagi2020ha,
  author = {Vishwanath A. Sindagi and Vishal M. Patel},
  title = {{HA-CCN: Hierarchical Attention-Based Crowd Counting Network}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=orrsCCQKHWo},
}
Figure