CNN-ViT Supported Weakly-Supervised Video Segment Level Anomaly Detection

Lei Jiao, Md. Haidar Sharif, Lei Jiao 0001, Christian W. OmlinSensors 2023 · DOI

TL;DR

This paper introduces CNN-ViT-TSAN, a novel framework for weakly-supervised video anomaly detection that combines the strengths of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). By leveraging pretrained models and a temporal self-attention network (TSAN) with a unique Mahalanobis distance-based scoring mechanism, the approach achieves superior performance in identifying unusual events in videos with minimal annotation effort.

What Problem Does This Paper Address?

Video anomaly event detection (VAED) is crucial for smart surveillance systems, aiming to identify unusual or unexpected events in video footage. Traditional VAED methods fall into three categories: fully supervised (requiring detailed frame-level annotations, which are costly and time-consuming to obtain), unsupervised (often underperforming due to limited understanding of anomalies), and weakly supervised (WVAED). WVAED offers a practical compromise by using only video-level labels (normal or abnormal), significantly reducing annotation costs. However, existing WVAED methods often rely heavily on the quality of pretrained feature extractors, and there's a need for more robust methods that can effectively capture both spatial and temporal information while handling the inherent ambiguity of weakly labeled data. This paper specifically addresses the challenge of improving WVAED performance by effectively integrating diverse feature representations and enhancing temporal modeling.

Methods

The core of this work is the CNN-ViT-TSAN framework, a generalized architecture for WVAED. It comprises several key components:

  • Feature Extraction: The framework utilizes pretrained feature extractors from both CNNs (specifically C3D and I3D) and Vision Transformers (specifically CLIP). This allows for capturing a rich set of visual representations.
  • Temporal Self-Attention Network (TSAN): This novel network models the continuity between video snippets and identifies the most relevant ones. It incorporates:
  • Temporal Scoring Module: Instead of relying on simple mean feature magnitudes, this module uses Mahalanobis distances to calculate anomaly scores for snippets. This is a key innovation, as Mahalanobis distance accounts for feature correlations and scaling, leading to more robust anomaly scoring.
  • Top-k Selecting Module: This module selects the k most significant snippets based on the scores generated by the temporal scoring module.
  • Multiplying-Averaging Module: This module processes the selected snippets to produce reweighed attention features.
  • Information Fusion: For models that combine CNN and ViT features (e.g., C3D-CLIP-TSAN), an information fusion module is used to combine these representations through element-wise addition.
  • Separation Maximization Supervisor (SMS): This component is used during training to maximize the separation between normal and abnormal snippets, further refining the model's ability to distinguish between them.
  • Loss Optimization: The framework uses a combination of score-based binary cross-entropy loss and the SMS loss to optimize the model.

The paper proposes five specific models within this framework: C3D-TSAN, I3D-TSAN, CLIP-TSAN, C3D-CLIP-TSAN, and I3D-CLIP-TSAN.

Datasets

The models were evaluated on several widely used benchmark datasets for video anomaly detection:

  • UMN Dataset: Features five staged videos with abnormal events characterized by people running in distinct directions.
  • UCSD-Peds Dataset: Consists of two sub-datasets (UCSD-Ped1 and UCSD-Ped2) capturing pedestrian anomalies on a walkway.
  • ShanghaiTech Dataset: A medium-scale dataset with scenes from multiple campus areas, containing both normal and anomalous videos.
  • UCF-Crime Dataset: A large-scale dataset with 1900 untrimmed videos covering 13 types of real-world anomalies.

The paper notes that while UMN, UCSD-Ped1, and UCSD-Ped2 are popular, they are relatively small. The ShanghaiTech and UCF-Crime datasets are considered large-scale and more challenging.

Key Insights and Ideas

  • Mahalanobis Distance for Anomaly Scoring: The introduction of Mahalanobis distance for scoring snippets is a significant departure from traditional mean-based approaches. It effectively captures complex relationships between features, leading to more accurate anomaly detection, especially in scenarios where simple magnitude differences are insufficient.
  • Synergy of CNN and ViT: Combining features from both CNNs (for spatial-temporal understanding) and ViTs (for rich contextual representations) proves to be more effective than using either alone. The information fusion mechanism plays a crucial role in leveraging this synergy.
  • TSAN for Temporal Modeling: The Temporal Self-Attention Network (TSAN) effectively models temporal dependencies and highlights important snippets, contributing to improved performance.
  • Weak Supervision is Practical: The success of the proposed WVAED models on various datasets reinforces the practicality and effectiveness of using video-level labels for anomaly detection, significantly reducing annotation burden.

Key Results

The proposed CNN-ViT-TSAN framework, particularly the I3D-CLIP-TSAN model, demonstrated superior performance across the evaluated datasets.

  • Performance Gains: The paper reports significant performance gains, with the Mahalanobis metric alone contributing to an average of 5% better performance compared to using mean snippet feature magnitudes across datasets.
  • AUC Scores: The I3D-CLIP-TSAN model achieved state-of-the-art or near state-of-the-art frame-level AUC scores on the ShanghaiTech and UCF-Crime datasets. For instance, on the UCF-Crime dataset, it achieved an AUC of 0.8897, outperforming many existing methods.
  • Generalizability: The models showed good generalizability, performing well on both simpler datasets like UMN and more complex ones like ShanghaiTech and UCF-Crime.
  • Statistical Significance: Nemenyi post hoc tests indicated that the I3D-CLIP-TSAN model was statistically superior or comparable to many existing methods, especially on the ShanghaiTech and UCF-Crime datasets.

Limitations and Future Work

  • Dependency on Pretrained Models: The performance of the WVAED models is inherently tied to the quality of the pretrained CNN and ViT feature extractors. This reliance can also make the training process computationally expensive.
  • Snippet Length Handling: In the testing phase, video clips shorter than the defined snippet length (∆=16 frames) were discarded or padded. The paper acknowledges that discarding these clips might lead to the loss of potentially useful information.
  • Future Work: The authors suggest exploring other ViT-based feature extractors beyond CLIP, such as VisualBERT, ViLBERT, and data-efficient CLIP. They also express interest in testing their models on newer benchmarks like the XD-Violence dataset, though accessibility issues currently prevent this.

Summary updated

BibTeX
@article{jiao2023cnn,
  author = {Lei Jiao and Md. Haidar Sharif and Lei Jiao and Christian W. Omlin},
  title = {{CNN-ViT Supported Weakly-Supervised Video Segment Level Anomaly Detection}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23187734},
}
Figure