Multi-Level Bottom-Top and Top-Bottom Feature Fusion for Crowd Counting

Vishwanath A. Sindagi, Vishal M. PatelICCV 2019 · OpenReview

Abstract

Crowd counting in congested scenes is a significant challenge due to extreme scale variations within and across images. While multi-scale feature fusion is a common approach, standard methods often fail to effectively combine features for highly congested environments. This paper introduces a novel architecture featuring Multi-Level Bottom-Top and Top-Bottom Fusion (MBTTBF) and Scale Complementary Feature Extraction Blocks (SCFB). By leveraging bidirectional information flow and cross-scale residual functions, the proposed network efficiently integrates spatial details and semantic context. Experimental results on ShanghaiTech, UCF_CC_50, and UCF-QNRF datasets demonstrate state-of-the-art performance.

Introduction

Estimating the number of people in congested scenes—such as marathons, protests, or festivals—is a critical task for public safety and urban planning. The primary difficulty lies in scale variation: a single image may contain individuals ranging from a few pixels to hundreds of pixels in size. Standard Convolutional Neural Networks (CNNs) struggle with this; shallow layers preserve spatial resolution but lack semantic context, while deeper layers capture high-level concepts but lose fine-grained detail. Previous attempts to bridge this gap via simple feature concatenation often fail to capture the complementary information necessary for precise counting in dense crowds.

Methodology

Multi-Level Bidirectional Fusion (MBTTBF)

The core innovation is the Multi-Level Bottom-Top and Top-Bottom Fusion (MBTTBF) scheme. Unlike traditional one-way fusion, this architecture employs a bidirectional hierarchy. The "bottom-top" path propagates fine-grained spatial details from shallower layers to deeper ones, while the "top-bottom" path sends semantic context back down to help shallower layers suppress background noise (e.g., trees or buildings that resemble heads).

Evolution of fusion strategies from simple CNNs (a) to the proposed multi-level bidirectional fusion (f).

Scale Complementary Feature Extraction Blocks (SCFB)

To make the fusion more effective, the authors introduce SCFB. These blocks utilize cross-scale residual functions to explicitly force the network to learn features that are missing or complementary between adjacent layers. This reduces redundancy and ensures that the fused features provide a more comprehensive representation of the crowd at various scales.

Detailed architecture of the MBTTBF network, highlighting the bidirectional fusion paths and the integration of SCFB blocks.

Scale-Aware Supervision

To improve training, the researchers developed a principled way to generate ground-truth density maps. Using a Markov Random Field (MRF) framework combined with superpixel segmentation, they estimate head sizes more accurately than standard nearest-neighbor heuristics. This provides a cleaner supervision signal, allowing the network to better learn scale-aware features.

Results and Discussion

The proposed MBTTBF-SCFB model was evaluated on several challenging benchmarks:

  • UCF-QNRF: Achieved a Mean Absolute Error (MAE) of 97.5, significantly outperforming previous state-of-the-art methods like CAN (107.0).
  • ShanghaiTech Part A: Reached an MAE of 60.2, surpassing established models such as CSRNet (68.2) and SANet (67.0).
  • Qualitative Performance: Visualizations indicate that the model produces cleaner density maps with fewer false positives in background areas compared to traditional fusion methods.

Conclusion

By rethinking how multi-scale features are fused, this work establishes a new standard for crowd counting in congested scenes. The combination of bidirectional multi-level fusion and scale-complementary blocks allows the network to effectively handle the extreme scale variations inherent in dense crowds. Future research may focus on optimizing the computational efficiency of these complex fusion paths for real-time deployment.

Summary updated

Abstract

Crowd counting presents enormous challenges in the form of large variation in scales within images and across the dataset. These issues are further exacerbated in highly congested scenes. Approaches based on straightforward fusion of multi-scale features from a deep network seem to be obvious solutions to this problem. However, these fusion approaches do not yield significant improvements in the case of crowd counting in congested scenes. This is usually due to their limited abilities in effectively combining the multi-scale features for problems like crowd counting. To overcome this, we focus on how to efficiently leverage information present in different layers of the network. Specifically, we present a network that involves: (i) a multi-level bottom-top and top-bottom fusion (MBTTBF) method to combine information from shallower to deeper layers and vice versa at multiple levels, (ii) scale complementary feature extraction blocks (SCFB) involving cross-scale residual functions to explicitly enable flow of complementary features from adjacent conv layers along the fusion paths. Furthermore, in order to increase the effectiveness of the multi-scale fusion, we employ a principled way of generating scale-aware ground-truth density maps for training. Experiments conducted on three datasets that contain highly congested scenes (ShanghaiTech, UCF_CC_50, and UCF-QNRF) demonstrate that the proposed method is able to outperform several recent methods in all the datasets.
BibTeX
@inproceedings{sindagi2019multi,
  author = {Vishwanath A. Sindagi and Vishal M. Patel},
  title = {{Multi-Level Bottom-Top and Top-Bottom Feature Fusion for Crowd Counting}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=Eghzn1TJznS},
}
Figure