A Two-Stage Approach to Important Area Detection in Gathering Place Using a Novel Multi-Input Attention Network

Jianqiang Xu, Haoyu Zhao, Weidong MinSensors 2021 · DOI

TL;DR

This paper introduces a novel two-stage approach for automatically detecting "important areas" in crowded gathering places, such as stages or event sites. The method first generates candidate areas using traditional image processing techniques and then employs a Multi-input Attention Network (MAN) to classify these candidates, effectively identifying regions that attract significant human attention.

What Problem Does This Paper Address?

In crowded environments, identifying "important areas" – regions that naturally draw people's attention and where crowds tend to gather – is crucial for security and crowd management. While humans can easily discern these areas based on visual cues and social behavior, traditional object detection methods struggle. These methods often focus on specific objects rather than the broader contextual importance of a region. Existing approaches also lack an efficient way to first identify potential important areas and then accurately assess whether they truly capture public attention. This paper aims to bridge this gap by developing a system that can automatically pinpoint these attention-grabbing zones.

Methods

The proposed solution is a two-stage pipeline:

  1. Important Area Candidate Generation: This initial stage uses a combination of classical image processing algorithms to identify potential important areas. These include:
  2. K-means++: For initial segmentation of foreground (people) and background.
  3. Image Dilation: To fill holes and smooth out noise in the segmented regions.
  4. Median Filtering: To further remove impulse noise and refine the image.
  5. RLSA (Run Length Smoothing Algorithm): To link neighboring black pixels and identify candidate regions. This stage aims to generate a set of candidate bounding boxes that are likely to contain important areas.

  6. Important Area Candidate Classification (Multi-input Attention Network - MAN): This stage takes the generated candidates and uses a novel neural network, MAN, to classify them. MAN is designed as a multi-input network that fuses:

  7. Interior Patch: The candidate area itself.
  8. Exterior Patch: The context surrounding the candidate area.
  9. Entire Image: The global scene information. MAN incorporates Channel Attention (C-A) and Spatial Attention (S-A) modules. These modules, inspired by CBAM, help the network focus on relevant features within the image patches, enhancing the representation of important areas by emphasizing both channel-wise and spatial information.

Datasets

  • Gathering Place Important Area Detection (GPIAD) Dataset: The authors introduce a new dataset specifically for this task. It comprises 1200 images of various gathering place scenes. The dataset includes 900 images for training and 300 for testing. Important areas are manually annotated from the candidates generated by the first stage.

Key Insights and Ideas

  • Context is King: The paper emphasizes that simply detecting objects isn't enough; understanding the surrounding context and how it influences human attention is key to identifying "important areas."
  • Two-Stage Approach Synergy: Combining traditional image processing for candidate generation with a sophisticated attention-based network for classification proves effective. The first stage efficiently narrows down possibilities, allowing the second stage to focus its computational power.
  • Attention for Importance: The novel MAN architecture, with its C-A and S-A modules, demonstrates the power of attention mechanisms in not just highlighting salient features but also in understanding the importance of a region based on human behavior (crowd gathering).
  • Domain-Specific Dataset: The creation of the GPIAD dataset is a significant contribution, providing a benchmark tailored to the specific problem of important area detection in crowded scenes, which existing general datasets might not adequately address.

Key Results

  • MAN Performance: The proposed MAN achieved a Top-1 accuracy of 81.27% and a Top-3 accuracy of 92.65% on the GPIAD dataset.
  • Comparison with SOTAs: MAN significantly outperformed established classification networks like VGGNet, ResNet, DenseNet, and MobileNet on the GPIAD dataset, achieving substantially higher Top-1 and Top-3 accuracies.
  • Comparison with Object Detectors: The proposed method (51.4% mAP) outperformed general object detection frameworks (BBC Net, M2Det, YOLOv4) in terms of mean Average Precision (mAP) on the GPIAD dataset, highlighting the limitations of general detectors for this specific task.
  • Ablation Studies: Ablation studies confirmed the effectiveness of both the C-A and S-A modules, with the full MAN architecture yielding the best performance. Removing both modules significantly degraded accuracy, similar to VGGNet's performance.

Limitations and Future Work

  • Exterior Patch Generation: The paper notes that generating exterior patches automatically during testing can be unstable due to complex crowd environments. Future work will focus on improving this automatic generation method.
  • Future Technologies: The authors suggest exploring newer technologies like Graph Convolutional Networks (GCNs) and Generative Adversarial Networks (GANs) to further enhance the method's capabilities.

Summary updated

Abstract

An important area in a gathering place is a region attracting the constant attention of people and has evident visual features, such as a flexible stage or an open-air show. Finding such areas can help security supervisors locate the abnormal regions automatically. The existing related methods lack an efficient means to find important area candidates from a scene and have failed to judge whether or not a candidate attracts people's attention. To realize the detection of an important area, this study proposes a two-stage method with a novel multi-input attention network (MAN). The first stage, called important area candidate generation, aims to generate candidate important areas with an image-processing algorithm (i.e., K-means++, image dilation, median filtering, and the RLSA algorithm). The candidate areas can be selected automatically for further analysis. The second stage, called important area candidate classification, aims to detect an important area from candidates with MAN. In particular, MAN is designed as a multi-input network structure, which fuses global and local image features to judge whether or not an area attracts people's attention. To enhance the representation of candidate areas, two modules (i.e., channel attention and spatial attention modules) are proposed on the basis of the attention mechanism. These modules are mainly based on multi-layer perceptron and pooling operation to reconstruct the image feature and provide considerably efficient representation. This study also contributes to a new dataset called gathering place important area detection for testing the proposed two-stage method. Lastly, experimental results show that the proposed method has good performance and can correctly detect an important area.
BibTeX
@inproceedings{xu2021two,
  author = {Jianqiang Xu and Haoyu Zhao and Weidong Min},
  title = {{A Two-Stage Approach to Important Area Detection in Gathering Place Using a Novel Multi-Input Attention Network}},
  booktitle = {Sensors},
  year = {2021},
  doi = {10.3390/s22010285},
}
Figure