Violence Detection Using Spatiotemporal Features with 3D Convolutional Neural Network

Fath U. Min Ullah, Amin Ullah, Khan Muhammad, Ijaz Ul Haq, Sung Wook Baik

TL;DR

This paper introduces a three-stage deep learning framework for violence detection in surveillance videos. By first efficiently detecting people to reduce processing load and then employing a 3D Convolutional Neural Network (CNN) to capture spatiotemporal features, the method achieves high accuracy. The system is further optimized for real-time deployment, outperforming existing state-of-the-art approaches on benchmark datasets.

What Problem Does This Paper Address?

The proliferation of surveillance cameras in smart cities generates vast amounts of video data, creating a need for automated systems to monitor for abnormal or violent activities. Manual monitoring is slow and prone to errors, leading to potential casualties and damages. Existing automated violence detection methods often struggle with complex visual patterns, inefficient processing of irrelevant frames, and limited accuracy on diverse datasets. This paper aims to address these challenges by proposing a more efficient and accurate deep learning-based violence detection system.

Methods

The proposed method employs a triple-staged, end-to-end deep learning framework:

  1. Person Detection (Pre-processing): To optimize processing, a lightweight MobileNet-SSD CNN model is used to detect persons in the video stream. This step filters out frames without people, significantly reducing the computational burden.
  2. Spatiotemporal Feature Extraction: A sequence of 16 frames containing detected persons is fed into a 3D Convolutional Neural Network (3D CNN). Unlike 2D CNNs that process frames independently, 3D CNNs can capture both spatial and temporal information, crucial for understanding motion and actions over time. The paper specifically fine-tunes a pre-trained C3D model.
  3. Violence Classification and Optimization: The extracted spatiotemporal features are passed to a Softmax classifier for predicting violent or non-violent activity. For real-time deployment, the trained 3D CNN model is optimized using Intel's OpenVINO toolkit, converting it into an intermediate representation (IR) for faster inference on end platforms.

Datasets

The proposed method was evaluated on three publicly available datasets:

  • Violent Crowd [14]: Contains 246 videos, split into 123 violent and 123 non-violent clips, with a resolution of 320x240 pixels.
  • Violence in Movies [7]: Consists of 200 video clips from action movies, with an average resolution of 360x250 pixels and 50 frames per clip.
  • Hockey Fight [7]: Comprises 1000 short video clips from the NHL, with 500 labeled as fight and 500 as non-fight, at a resolution of 360x288 pixels and 50 frames per clip.

The paper also mentions testing the model trained on one dataset against others, providing insights into its generalization capabilities.

Key Insights and Ideas

  • Efficient Pre-processing is Crucial: By first detecting persons, the system avoids processing large portions of video that contain no human activity, drastically improving efficiency and reducing memory usage.
  • 3D CNNs for Spatiotemporal Understanding: The paper highlights the advantage of 3D CNNs over 2D CNNs for action recognition tasks like violence detection, as they can inherently capture temporal dynamics within video sequences.
  • Transfer Learning and Fine-tuning: The use of a pre-trained C3D model and fine-tuning it on specific violence detection datasets is a key strategy for achieving high accuracy, leveraging knowledge learned from larger, general-purpose video datasets.
  • Model Optimization for Deployment: The integration of OpenVINO for model optimization is a practical step towards making the system deployable in real-world surveillance scenarios where computational resources might be limited.

Key Results

The proposed method demonstrated strong performance across the evaluated datasets:

  • Violence in Movies: Achieved the highest accuracy of 99.9%.
  • Violent Crowd: Achieved an accuracy of 98%.
  • Hockey Fight: Achieved an accuracy of 96%.

The paper also shows that the proposed method outperforms several existing state-of-the-art methods on these datasets, as detailed in Table 8. The ROC curves and AUC values (ranging from 0.970 to 0.997) further indicate the model's robust discriminative power.

Limitations and Future Work

  • Limitations: While the paper reports high accuracies, it's worth noting that some comparisons in Table 8 show accuracies for other methods that are very close or even slightly higher in specific instances (e.g., 99.5% for Improved Fisher Vectors on Violence in Movies). The paper also mentions that accuracy on the Violent Crowd dataset was lower (54%) when the model trained on Violence in Movies was tested, suggesting potential generalization challenges.
  • Future Work: The authors plan to implement the system on resource-constrained devices and explore edge intelligence for violence recognition in IoT environments to enable quicker responses.

Summary updated

Abstract

The worldwide utilization of surveillance cameras in smart cities has enabled researchers to analyze a gigantic volume of data to ensure automatic monitoring. An enhanced security system in smart cities, schools, hospitals, and other surveillance domains is mandatory for the detection of violent or abnormal activities to avoid any casualties which could cause social, economic, and ecological damages. Automatic detection of violence for quick actions is very significant and can efficiently assist the concerned departments. In this paper, we propose a triple-staged end-to-end deep learning violence detection framework. First, persons are detected in the surveillance video stream using a light-weight convolutional neural network (CNN) model to reduce and overcome the voluminous processing of useless frames. Second, a sequence of 16 frames with detected persons is passed to 3D CNN, where the spatiotemporal features of these sequences are extracted and fed to the Softmax classifier. Furthermore, we optimized the 3D CNN model using an open visual inference and neural networks optimization toolkit developed by Intel, which converts the trained model into intermediate representation and adjusts it for optimal execution at the end platform for the final prediction of violent activity. After detection of a violent activity, an alert is transmitted to the nearest police station or security department to take prompt preventive actions. We found that our proposed method outperforms the existing state-of-the-art methods for different benchmark datasets.
BibTeX
@inproceedings{ullah2019violence,
  author = {Fath U. Min Ullah and Amin Ullah and Khan Muhammad and Ijaz Ul Haq and Sung Wook Baik},
  title = {{Violence Detection Using Spatiotemporal Features with 3D Convolutional Neural Network}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=jkKEotT0b9r},
}
Figure