A crowd clustering prediction and captioning technique for public health emergencies

Xiaoling Zhou, Guiping ZhuPeerJ Comput. Sci 2023 · DOI

TL;DR

This paper introduces a novel technique for simultaneously predicting crowd clustering and generating news captions for public health emergencies. By combining an improved multi-scale convolutional neural network (MCNN) for crowd analysis with a double-LSTM model for scene captioning, the authors aim to provide rapid and effective information for news reporting and social issue management.

What Problem Does This Paper Address?

Public health emergencies, like the COVID-19 pandemic, often lead to significant public gatherings and generate a large volume of news and social commentary. Effectively understanding and reporting on these events requires quickly identifying and analyzing crowd behavior. Specifically, this paper tackles two key challenges: 1) accurately predicting crowd density and clustering in real-time, even under difficult conditions like occlusion or varying crowd densities, and 2) automatically generating concise and informative news captions that capture the essence of these crowd events. Existing methods often focus on one aspect or struggle with the dynamic nature of crowd behavior and the need for rapid information dissemination.

Methods

The proposed method consists of two main components:

  • Crowd Clustering Prediction: The authors employ an "improved multi-scale convolutional neural network" (MCNN). This network utilizes multiple convolutional kernels of varying sizes (e.g., 11x11, 9x9, 7x7) to capture features at different scales and perspectives. To enhance receptive field and efficiency, they use superimposed smaller kernels (e.g., two 1x9 and 9x1 kernels to mimic a 9x9 kernel). The network first computes a density map of the crowd by identifying pedestrian heads and then uses this to predict crowd density. Beyond density, they also analyze crowd movement status using three eigenvalues: average kinetic energy, crowd density value, and crowd distribution entropy. These eigenvalues help detect abnormal crowd dispersion events.
  • Scene Captioning: For generating news text, a "double-LSTM" model is proposed. This model takes the scene information (likely derived from the crowd analysis) and uses an attention mechanism (Word Generation Attention - WGA) to focus on relevant parts of the scene. A language LSTM then processes this attended information to generate a sequence of words, forming a caption for the event.

Datasets

  • ShanghaiTech Dataset: Used for training and evaluating the crowd clustering prediction model. This dataset contains labeled images of crowds, with Part_A featuring denser crowds and Part_B sparser crowds.
  • UMN Dataset: Used for analyzing crowd movement patterns, specifically focusing on videos of abnormal crowd activity (random burst and same-direction burst).
  • MSCOCO Dataset: Used to evaluate the scene captioning model.

Key Insights and Ideas

  • Multi-scale CNN for Robust Crowd Analysis: The use of multi-scale convolutional kernels and superimposed smaller kernels is crucial for handling variations in crowd density, scale, and occlusion, leading to more accurate crowd counting and density estimation.
  • Crowd Movement Eigenvalues for Anomaly Detection: Analyzing kinetic energy, density, and distribution entropy provides a comprehensive way to detect and characterize abnormal crowd dispersion events, which are often precursors to social issues.
  • Double-LSTM with Attention for Contextual Captioning: The double-LSTM architecture with a WGA mechanism allows the model to effectively focus on salient scene features, enabling the generation of more relevant and informative news captions.
  • Integrated Approach for Public Health Emergencies: The paper highlights the value of integrating crowd analysis with natural language generation to provide timely and actionable information during public health crises.

Key Results

  • Crowd Clustering: The proposed method shows significant improvements in reducing Mean Absolute Error (MAE) and Mean Squared Error (MSE) on the ShanghaiTech dataset compared to existing models like MCNN, MSCNN, and CMTL. For instance, on Part_A, MAE and MSE were reduced by 22.0 and 35.7 respectively compared to MCNN. On Part_B, the method achieved the lowest MAE and MSE among all compared methods (7.7 and 12.4 respectively).
  • Crowd Movement Analysis: The analysis of UMN dataset videos demonstrated that the three eigenvalues (kinetic energy, density, entropy) effectively capture the dynamics of crowd dispersion, showing sharp increases in kinetic energy and entropy, and a decrease in density, at the onset of abnormal events.
  • Scene Captioning: The proposed double-LSTM model achieved competitive results on the MSCOCO dataset, particularly excelling in Cider scores (C), indicating its ability to generate descriptive and relevant captions.

Limitations and Future Work

  • Limitations: The paper notes that while their method performs well, it "lags slightly behind on Part_A" compared to TDF-CNN in crowd counting, though it outperforms TDF-CNN on Part_B. This suggests potential areas for further refinement in handling very dense crowd scenarios. The current work focuses on indoor abnormal crowd activity; extending to diverse outdoor environments could be a limitation.
  • Future Work: The authors suggest future work will involve studying the "expression of crowd action content" and directly identifying it to improve the accuracy of press release generation. This implies a deeper understanding of crowd dynamics and their translation into more nuanced textual descriptions.

Summary updated

BibTeX
@article{zhou2023crowd,
  author = {Xiaoling Zhou and Guiping Zhu},
  title = {{A crowd clustering prediction and captioning technique for public health emergencies}},
  journal = {PeerJ Comput. Sci.},
  year = {2023},
  doi = {10.7717/peerj-cs.1283},
}
Figure