SOFA-Net: Second-Order and First-order Attention Network for Crowd Counting

Haoran Duan, Shidong Wang, Yu Guan

Abstract

Automated crowd counting from images/videos has attracted more attention in recent years because of its wide application in smart cities. But modelling the dense crowd heads is challenging and most of the existing works become less reliable. To obtain the appropriate crowd representation, in this work we proposed SOFA-Net (Second-Order and First-order Attention Network): second-order statistics were extracted to retain selectivity of the channel-wise spatial information for dense heads while first-order statistics, which can enhance the feature discrimination for the heads' areas, were used as complementary information. Via a multi-stream architecture, the proposed second/first-order statistics were learned and transformed into attention for robust representation refinement. We evaluated our method on four public datasets and the performance reached state-of-the-art on most of them. Extensive experiments were also conducted to study the components in the proposed SOFA-Net, and the results suggested the high-capability of second/first-order statistics on modelling crowd in challenging scenarios.

Introduction

Crowd counting is a deceptively simple task that becomes a nightmare at scale. When analyzing high-density scenes, such as drone shots of music festivals or busy city squares, individual heads appear as tiny clusters of pixels, often heavily occluded or blurred. Traditional Convolutional Neural Networks (CNNs) often struggle in these environments, tending to underestimate dense areas and overestimate sparse ones.

Standard CNNs face a "dynamic range" problem. In high-density regions, features for individual heads overlap so significantly that the network loses "selectivity" — the ability to distinguish where one head ends and another begins. Conversely, in sparse areas, background noise is frequently mistaken for heads. The authors of SOFA-Net argue that standard "first-order" features (raw CNN activations) are insufficient to represent the complex spatial relationships inherent in dense crowds.

The SOFA-Net Architecture

To address these limitations, the researchers introduced SOFA-Net, an architecture built on a VGG16 backbone that incorporates dual-attention mechanisms. The "SOFA" acronym stands for Second-Order and First-order Attention.

The overall architecture of SOFA-Net, illustrating the multi-stream feature learning and statistical attention modules.

The architecture consists of several novel components:

  1. Second-Order Statistical Attention: This module calculates a covariance matrix of the feature maps. While first-order statistics identify where a feature is, second-order statistics capture the correlation between different feature channels across the image. This helps the model retain spatial selectivity, allowing it to "see" the distinctness of heads within a dense cluster.
  2. First-Order Statistical Attention: This acts as a complement, focusing on discriminating between head and non-head areas by emphasizing edges and textures.
  3. Statistic-Wise Convolution (SWC): To process these statistics, the authors designed a custom operation consisting of Independent Statistical Learning (ISL) and Cross-Channel Statistical Learning (CsL), transforming raw statistical summaries into meaningful attention masks.
  4. Multi-Stream Architecture & Normalization Mask: The network processes three streams (First-order, Second-order, and raw VGG features) and fuses them. It learns an unnormalized density map and a normalization mask separately. Multiplying these results in a refined final density map with significantly fewer artifacts in background areas.

Key Insights and Ablation Analysis

A critical finding of the study is that first and second-order statistics provide complementary, non-redundant information. First-order statistics are highly effective for discrimination (distinguishing heads from background), while second-order statistics excel at modeling the internal structure of dense clusters.

Visual comparison of ablation studies showing how the combination of first and second-order attention significantly sharpens the resulting density map.

As shown in the ablation study, removing either attention mechanism leads to increased error and blurrier density maps. The baseline model often produces diffuse heatmaps, whereas the full SOFA-Net (Second + First) identifies "likelihood peaks" at the center of each head, effectively fixing the "blurry" problem common in standard regression methods.

Experimental Results

SOFA-Net was evaluated on four standard benchmarks, achieving state-of-the-art (SOTA) or near-SOTA performance measured by Mean Absolute Error (MAE) and Mean Squared Error (MSE):

  • UCF_QNRF: Achieved an MAE of 96.2, outperforming the previous best (MBTTBF) at 97.5.
  • UCF_CC_50: Showed a massive improvement, reaching an MAE of 185 compared to the previous SOTA of 188.4 (L2SM).
  • ShanghaiTech (Part A & B): On the highly congested Part A, it reached an MAE of 57.5, a significant jump over SPANet (59.4).

Conclusion

SOFA-Net demonstrates that looking beyond simple feature activations to higher-order statistics is key to mastering dense crowd counting. While the model provides the most significant benefits in extremely congested scenarios, it remains competitive in sparser scenes. Future work may involve deeper theoretical exploration of the mathematical underpinnings of why second-order features are so effective for crowd representation.

Summary updated

BibTeX
@inproceedings{duan2020sofa,
  author = {Haoran Duan and Shidong Wang and Yu Guan},
  title = {{SOFA-Net: Second-Order and First-order Attention Network for Crowd Counting}},
  booktitle = {BMVC},
  year = {2020},
  url = {https://openreview.net/forum?id=vsCobACvxN},
}
Figure