Privacy-Preserving Video Anomaly Detection: A Survey

Yang Liu, Siao Liu, Xiaoguang Zhu, Jielin Li, Hao Yang, Liangyu Teng, Juncen Guo, Yan Wang, Dingkang Yang, Jing Liu

TL;DR

This paper presents the first systematic survey of Privacy-Preserving Video Anomaly Detection (P2VAD), a field dedicated to detecting dangerous events in surveillance without compromising individual privacy. The authors introduce a comprehensive taxonomy that categorizes methods across the entire data lifecycle—from hardware acquisition to distributed system deployment—providing a roadmap for building ethical and trustworthy AI surveillance.

What Problem Does This Paper Address?

Video Anomaly Detection (VAD) is essential for public safety, helping to identify accidents, thefts, or violence in real-time. However, traditional VAD relies on raw RGB video, which captures personally identifiable information (PII) such as faces, clothing, and specific environmental layouts. This creates significant ethical concerns, risks of data misuse, and legal hurdles (like GDPR compliance) that prevent widespread adoption.

Current research in P2VAD is fragmented. Some researchers focus on hardware (like infrared cameras), others on features (like skeletons), and others on secure computing (like federated learning). This paper unifies these disparate efforts into a single, cohesive framework to guide future research toward systems that are both effective and privacy-compliant.

Methods

The authors propose a taxonomy organized into three critical stages of the data lifecycle, as shown in Figure 1:

Fig. 1. Taxonomy of Privacy-Preserving Video Anomaly Detection (P2VAD).
  1. Non-Identifiable Elements (NIE): These methods alter the data acquisition stage to avoid capturing sensitive info from the start. This includes:
  2. Encrypted Video Coding: Processing compressed bitstreams (H.264/HEVC) that are unreadable to humans but contain motion vectors usable by AI.
  3. Sensitive Area Masking: Using object detection to black out human figures.
  4. Non-Visible Light Cameras (NVLCs): Using infrared, depth, or event cameras that capture motion and heat signatures without revealing facial details.
  5. Desensitized Intermediate Modalities (DIM): These focus on the model learning stage by extracting privacy-neutral representations from RGB video.
  6. Skeleton-based VAD: Extracting human joint keypoints. This captures the "action" while discarding appearance.
  7. Optical Flow: Using pixel-level motion vectors to focus on dynamics while ignoring texture and color.
  8. Edge-Cloud Intelligence (ECI): These address privacy during system application and deployment.
  9. Privacy Computing: Using techniques like Homomorphic Encryption (computing on encrypted data) and Differential Privacy (adding noise to prevent reverse-engineering).
  10. Federated Learning (FL): Training a global model across multiple local devices without ever sharing the raw video data between them.

Datasets

The survey categorizes datasets into two main groups:

  • Generalized VAD Datasets: Traditional RGB datasets like UCF-Crime, ShanghaiTech, CUHK Avenue, and UCSD Ped1/Ped2. These are frequently repurposed for P2VAD by extracting skeletons or optical flow.
  • Specialized P2VAD Datasets: Datasets designed specifically for privacy research.
  • UBnormal: A large-scale synthetic dataset using computer-generated avatars in real backgrounds to avoid using real people.
  • PHEVA: A human-centric dataset focused on skeleton keypoints.
  • Anonymized UCF-Crime: A version of the popular crime dataset where sensitive PII has been masked using U-Net architectures.

Key Insights and Ideas

  • Motion is the Message: A core insight is that most real-world anomalies are motion-related (e.g., a fall, a fight, a car crash). Because appearance information often contributes little to detection but carries the most privacy risk, "desensitizing" the data by focusing on skeletons or motion vectors is a highly effective strategy.
  • Hardware as a First Line of Defense: Moving privacy protection to the hardware level (e.g., using depth sensors or event cameras) is more secure than trying to redact sensitive information after it has already been captured in high-definition RGB.
  • The System-Level Gap: Privacy isn't just a data format problem; it's a pipeline problem. Even if you use skeletons, the transmission of those features across a network can be a point of failure, necessitating ECI-level protections like federated learning.

Key Results

The paper summarizes quantitative advantages across different P2VAD pathways:

  • Efficiency: Encrypted video methods can achieve massive speedups, with some models processing over 1250 fps by avoiding the need to fully decode video frames.
  • Privacy vs. Performance: Anonymization techniques (like TeD-SPAD) have shown they can reduce the ability to predict private attributes by 32.25% while only suffering a minor 3.69% drop in detection accuracy (AUC).
  • Robustness: Skeleton-based methods (DIM) are often more robust to environmental changes like lighting and weather compared to RGB-based methods, proving that privacy-preserving features can actually improve model stability in the real world.

Limitations and Future Work

  • Context Loss: Stripping appearance can sometimes remove vital context. For example, a person running is normal at a track but anomalous in a bank; without seeing the background or clothing, the AI might struggle.
  • Computational Overhead: Cryptographic methods like Homomorphic Encryption and the communication requirements of Federated Learning often exceed the capabilities of low-power edge devices (smart cameras).
  • The Generative AI Challenge: The rise of Large Language Models (LLMs) and Generative AI offers new ways to describe anomalies, but these models tend to describe everything they see, potentially creating new privacy leaks that current P2VAD frameworks aren't yet equipped to handle.

Summary updated

Abstract

Video Anomaly Detection (VAD) aims to automatically analyze spatiotemporal patterns in surveillance videos collected from open spaces to detect anomalous events that may cause harm, such as fighting, stealing, and car accidents. However, vision-based surveillance systems such as closed-circuit television often capture personally identifiable information. The lack of transparency and interpretability in video transmission and usage raises public concerns about privacy and ethics, limiting the real-world application of VAD. Recently, researchers have focused on privacy concerns in VAD by conducting systematic studies from various perspectives including data, features, and systems, making Privacy-Preserving Video Anomaly Detection (P2VAD) a hotspot in the AI community. However, current research in P2VAD is fragmented, and prior reviews have mostly focused on methods using RGB sequences, overlooking privacy leakage and appearance bias considerations. To address this gap, this article is the first to systematically reviews the progress of P2VAD, defining its scope and providing an intuitive taxonomy. We outline the basic assumptions, learning frameworks, and optimization objectives of various approaches, analyzing their strengths, weaknesses, and potential correlations. Additionally, we provide open access to research resources such as benchmark datasets and available code. Finally, we discuss key challenges and future opportunities from the perspectives of AI development and P2VAD deployment, aiming to guide future work in the field.
BibTeX
@misc{liu2024privacy,
  author = {Yang Liu and Siao Liu and Xiaoguang Zhu and Jielin Li and Hao Yang and Liangyu Teng and Juncen Guo and Yan Wang and Dingkang Yang and Jing Liu},
  title = {{Privacy-Preserving Video Anomaly Detection: A Survey}},
  year = {2024},
  eprint = {2411.14565},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.14565},
}
Figure