TL;DR
This paper introduces a three-stage deep learning framework for violence detection in surveillance videos. By first efficiently detecting people to reduce processing load and then employing a 3D Convolutional Neural Network (CNN) to capture spatiotemporal features, the method achieves high accuracy. The system is further optimized for real-time deployment, outperforming existing state-of-the-art approaches on benchmark datasets.
What Problem Does This Paper Address?
The proliferation of surveillance cameras in smart cities generates vast amounts of video data, creating a need for automated systems to monitor for abnormal or violent activities. Manual monitoring is slow and prone to errors, leading to potential casualties and damages. Existing automated violence detection methods often struggle with complex visual patterns, inefficient processing of irrelevant frames, and limited accuracy on diverse datasets. This paper aims to address these challenges by proposing a more efficient and accurate deep learning-based violence detection system.
Methods
The proposed method employs a triple-staged, end-to-end deep learning framework:
- Person Detection (Pre-processing): To optimize processing, a lightweight MobileNet-SSD CNN model is used to detect persons in the video stream. This step filters out frames without people, significantly reducing the computational burden.
- Spatiotemporal Feature Extraction: A sequence of 16 frames containing detected persons is fed into a 3D Convolutional Neural Network (3D CNN). Unlike 2D CNNs that process frames independently, 3D CNNs can capture both spatial and temporal information, crucial for understanding motion and actions over time. The paper specifically fine-tunes a pre-trained C3D model.
- Violence Classification and Optimization: The extracted spatiotemporal features are passed to a Softmax classifier for predicting violent or non-violent activity. For real-time deployment, the trained 3D CNN model is optimized using Intel's OpenVINO toolkit, converting it into an intermediate representation (IR) for faster inference on end platforms.
Datasets
The proposed method was evaluated on three publicly available datasets:
- Violent Crowd [14]: Contains 246 videos, split into 123 violent and 123 non-violent clips, with a resolution of 320x240 pixels.
- Violence in Movies [7]: Consists of 200 video clips from action movies, with an average resolution of 360x250 pixels and 50 frames per clip.
- Hockey Fight [7]: Comprises 1000 short video clips from the NHL, with 500 labeled as fight and 500 as non-fight, at a resolution of 360x288 pixels and 50 frames per clip.
The paper also mentions testing the model trained on one dataset against others, providing insights into its generalization capabilities.
Key Insights and Ideas
- Efficient Pre-processing is Crucial: By first detecting persons, the system avoids processing large portions of video that contain no human activity, drastically improving efficiency and reducing memory usage.
- 3D CNNs for Spatiotemporal Understanding: The paper highlights the advantage of 3D CNNs over 2D CNNs for action recognition tasks like violence detection, as they can inherently capture temporal dynamics within video sequences.
- Transfer Learning and Fine-tuning: The use of a pre-trained C3D model and fine-tuning it on specific violence detection datasets is a key strategy for achieving high accuracy, leveraging knowledge learned from larger, general-purpose video datasets.
- Model Optimization for Deployment: The integration of OpenVINO for model optimization is a practical step towards making the system deployable in real-world surveillance scenarios where computational resources might be limited.
Key Results
The proposed method demonstrated strong performance across the evaluated datasets:
- Violence in Movies: Achieved the highest accuracy of 99.9%.
- Violent Crowd: Achieved an accuracy of 98%.
- Hockey Fight: Achieved an accuracy of 96%.
The paper also shows that the proposed method outperforms several existing state-of-the-art methods on these datasets, as detailed in Table 8. The ROC curves and AUC values (ranging from 0.970 to 0.997) further indicate the model's robust discriminative power.
Limitations and Future Work
- Limitations: While the paper reports high accuracies, it's worth noting that some comparisons in Table 8 show accuracies for other methods that are very close or even slightly higher in specific instances (e.g., 99.5% for Improved Fisher Vectors on Violence in Movies). The paper also mentions that accuracy on the Violent Crowd dataset was lower (54%) when the model trained on Violence in Movies was tested, suggesting potential generalization challenges.
- Future Work: The authors plan to implement the system on resource-constrained devices and explore edge intelligence for violence recognition in IoT environments to enable quicker responses.
Summary updated