Safety After Dark: A Privacy Compliant and Real-Time Edge Computing Intelligent Video Analytics for Safer Public Transportation

Johan Barthélemy, Umair Iqbal, Yan Qian, Mehrdad Amirghasemi, Pascal PerezSensors 2024 · DOI

TL;DR

This paper introduces "Safety After Dark," an AI-powered video analytics system for real-time detection of unsafe behaviors on public transport. By leveraging edge computing and a novel Temporal Pyramid Network (TPN) model trained on a custom dataset, the system aims to enhance passenger safety and operational efficiency in train stations.

What Problem Does This Paper Address?

Public transportation systems are critical infrastructure, but they are increasingly vulnerable to security threats, particularly acts of violence and harassment. Traditional security methods, relying on human monitoring of CCTV feeds, are often labor-intensive, prone to errors, and struggle with real-time response, especially in large and complex environments. This gap in real-time, automated detection of unsafe behaviors poses a significant risk to passenger safety and the smooth operation of transit networks. The paper aims to bridge this gap by developing an intelligent, privacy-compliant, and efficient AI solution.

Methods

The core of the "Safety After Dark" system is a Temporal Pyramid Network (TPN) action recognition model. TPN is designed to capture temporal dynamics at various scales, making it effective for recognizing actions with both rapid movements and longer sequences. The system employs an end-to-end pipeline that includes:

  • CCTV Video Capture: Streams from existing train station CCTV networks.
  • Secure Streaming: AWS DirectConnect is used for secure and reliable transmission of video feeds to a local edge server.
  • Edge Computing: A local edge server with NVIDIA GPUs processes the video data. This approach minimizes latency and bandwidth requirements by performing inference locally.
  • DeepStream SDK: NVIDIA's DeepStream SDK is utilized for efficient video decoding, frame batching, and integration with AI models.
  • ONNXRuntime: This inference engine enables fast, cross-platform deployment of the TPN model on the edge server.
  • TPN Model: The Temporal Pyramid Network is the AI model responsible for identifying unsafe behaviors. It was trained on a custom dataset.
  • MQTT Protocol: Used for lightweight and reliable transmission of inference results (alerts) from the edge server to a web application.
  • Web Application: A user-friendly interface for real-time monitoring, alert confirmation, and data analysis.
  • Privacy Features: An optional face-blurring module is included to enhance privacy compliance.

Datasets

The paper introduces a newly curated video dataset called the Safety After Dark (SAD) dataset. This dataset was created by combining:

  • Open-source resources: Video clips from existing datasets like UCF-Crime and NTU CCTV-Fights Dataset, focusing on physical altercations.
  • Real-world footage: Provided by Sydney Trains, capturing authentic scenes.
  • Simulated trials: Two simulation exercises were conducted with university staff to enact various fight scenarios.
  • Routine activities: Video clips from datasets like UCF101 and VIRAT Video Dataset, depicting everyday actions (walking, sitting, etc.) to represent the "Normal" class.

The SAD dataset contains 2059 clips, with 1353 labeled as "Fighting" and 706 as "Normal." This dataset was crucial for training the TPN model, especially for addressing the domain shift problem observed with pre-trained models.

Key Insights and Ideas

  • Edge Computing is Key for Real-Time Public Transport Security: By processing video data locally on edge servers, the system achieves low latency and reduces reliance on constant cloud connectivity, making it practical for real-time applications in environments like train stations.
  • Custom Datasets are Essential for Domain-Specific AI: The success of the TPN model was significantly boosted by training on the custom SAD dataset, which included both real-world and simulated public transport scenarios. This highlights the limitations of relying solely on general-purpose datasets for specialized tasks.
  • Simulations Bridge the Data Gap and Improve Robustness: Conducting simulated fight scenarios was instrumental in generating diverse data, which directly led to improved model accuracy and a reduction in false positives, demonstrating the value of controlled data augmentation.
  • Continuous Retraining is Vital for Adaptability: The paper emphasizes the importance of a continuous retraining loop, where new data (including from false positives) is fed back to the model to improve its performance over time and adapt to evolving scenarios.
  • Privacy-Compliant Design is Crucial: The inclusion of features like optional face blurring demonstrates a commitment to ethical AI deployment in public spaces.

Key Results

  • Base Model Performance: The TPN model trained on open-source data achieved a validation accuracy of 93%.
  • Retrained Model Performance: After incorporating simulated data, the validation accuracy improved to 97%.
  • On-Site Deployment: During an 8-week live trial at Wollongong Train Station, the system achieved a reliable false-positive (FP) rate of 23%.
  • Incident Detection: The system correctly identified 30 true-positive incidents.
  • False Negatives: Six false negatives (missed incidents) were observed, particularly during rainy weather, highlighting a need for more data in adverse conditions.
  • FP Reduction: The retrained model significantly reduced FP rates compared to the base model, from an initial high rate to between 3-12 per hour per CCTV feed after deployment.

Limitations and Future Work

  • Adverse Weather Conditions: The system struggled with rainy weather, leading to a higher FP rate. This indicates a need for more training data that specifically captures incidents in such conditions.
  • Data Scarcity for Specific Scenarios: Obtaining sufficient training data for all types of unsafe behaviors and environmental conditions remains a challenge.
  • Continuous Improvement: The authors suggest exploring more advanced models like transformer-based action recognition models (e.g., AcT) or visual language models (VLMs).
  • Pose Estimation: Investigating models based on pose estimation (using 3D skeletons) could offer a more nuanced understanding of subtle behaviors.
  • System Robustness: Future work could focus on enhancing resilience to video stream interruptions and network failures, potentially through local buffering and multi-path streaming.
  • Full DeepStream Integration: Utilizing a complete DeepStream implementation for inference could further optimize GPU utilization.

Summary updated

Abstract

Public transportation systems play a vital role in modern cities, but they face growing security challenges, particularly related to incidents of violence. Detecting and responding to violence in real time is crucial for ensuring passenger safety and the smooth operation of these transport networks. To address this issue, we propose an advanced artificial intelligence (AI) solution for identifying unsafe behaviours in public transport. The proposed approach employs deep learning action recognition models and utilises technologies like NVIDIA DeepStream SDK, Amazon Web Services (AWS) DirectConnect, local edge computing server, ONNXRuntime and MQTT to accelerate the end-to-end pipeline. The solution captures video streams from remote train stations closed circuit television (CCTV) networks, processes the data in the cloud, applies the action recognition model, and transmits the results to a live web application. A temporal pyramid network (TPN) action recognition model was trained on a newly curated video dataset mixing open-source resources and live simulated trials to identify the unsafe behaviours. The base model was able to achieve a validation accuracy of 93% when trained using open-source dataset samples and was improved to 97% when live simulated dataset was included during the training. The developed AI system was deployed at Wollongong Train Station (NSW, Australia) and showcased impressive accuracy in detecting violence incidents during an 8-week test period, achieving a reliable false-positive (FP) rate of 23%. While the AI correctly identified 30 true-positive incidents, there were 6 cases of false negatives (FNs) where violence incidents were missed during the rainy weather suggesting more data in the training dataset related to bad weather. The AI model's continuous retraining capability ensures its adaptability to various real-world scenarios, making it a valuable tool for enhancing safety and the overall passenger experience in public transport settings.
BibTeX
@inproceedings{barthelemy2024safety,
  author = {Johan Barthélemy and Umair Iqbal and Yan Qian and Mehrdad Amirghasemi and Pascal Perez},
  title = {{Safety After Dark: A Privacy Compliant and Real-Time Edge Computing Intelligent Video Analytics for Safer Public Transportation}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24248102},
}
Figure