OE-CTST: Outlier-Embedded Cross Temporal Scale Transformer for Weakly-Supervised Video Anomaly Detection

Snehashis Majhi, Rui Dai, Quan Kong, Lorenzo Garattoni, Gianpiero Francesca, François BrémondWACV 2024

TL;DR

This paper introduces OE-CTST, a novel transformer-based framework for weakly-supervised video anomaly detection. It tackles the challenge of detecting both short and long anomalies by learning "anomaly-aware" temporal position embeddings and employing a multi-scale temporal transformer to capture diverse temporal relationships.

What Problem Does This Paper Address?

Detecting anomalies in real-world videos is a critical but challenging task, especially when dealing with untrimmed footage where precise temporal annotations are scarce. Existing methods struggle with two main issues: first, the difficulty in distinguishing between short, abrupt anomalies (like explosions) and long, gradual ones (like robberies), which exhibit different temporal cues. Second, the lack of precise temporal labels in weakly-supervised settings hinders the effective modeling of temporal dynamics. This paper aims to bridge this gap by developing a model that can effectively learn from video-level labels and accurately identify anomalies of varying durations.

Methods

The proposed OE-CTST framework consists of four main components:

  • Visual Encoder: This module extracts spatio-temporal features from video segments using a pre-trained Video-Swin transformer.
  • Outlier Embedder (OE): This is a novel component that learns temporal regularity from normal video segments. It generates "anomaly-aware" temporal position embeddings by identifying deviations from learned normal patterns. This helps the transformer better understand the temporal context of anomalies.
  • Cross Temporal Scale Transformer (CTST): This module leverages a multi-stage design with a "Cross Temporal Field Attention" (CTFA) block. It processes features at different temporal scales, allowing it to capture fine-grained temporal relations for short anomalies at lower levels and broader contextual relations for long anomalies at higher levels. CTFA specifically focuses on encoding correlations between temporally neighboring and distant tokens.
  • Detector: A simple MLP that takes the output from the CTST and assigns anomaly scores to each temporal token.

The training process involves two distinct batches: one for normal (one-class) data and one for mixed (normal + anomaly) data. The OE is trained on the one-class data, while the CTST and detector are trained on both.

Datasets

The proposed OE-CTST is evaluated on three publicly available datasets:

  • UCF-Crime: A widely used dataset for anomaly detection.
  • XD-Violence: Another popular dataset for video anomaly detection.
  • IITB-Corridor: A dataset focusing on anomalies in corridor surveillance scenarios.

The authors also utilize K-Fold cross-validation on UCF-Crime and IITB-Corridor to provide a more robust evaluation, especially for long-short anomaly performance.

Key Insights and Ideas

  • Anomaly-Aware Position Embeddings: The core innovation is the "Outlier Embedder," which moves beyond standard positional encodings. By learning temporal regularity from normal data, it can generate embeddings that highlight deviations, effectively guiding the transformer to focus on anomalous segments.
  • Multi-Scale Temporal Modeling: The CTST's multi-stage design with CTFA is crucial for handling the diverse temporal characteristics of short and long anomalies. This hierarchical approach allows for capturing both abrupt changes and gradual progressions.
  • CTFA for Cross-Temporal Correlations: The CTFA block's ability to model correlations between both nearby and distant temporal tokens is key to its effectiveness in capturing global temporal relations, which is essential for detecting anomalies of varying lengths.
  • Weak Supervision Advantage: The framework is designed for weakly-supervised settings, making it more practical for real-world scenarios where precise frame-level annotations are often unavailable.

Key Results

The OE-CTST demonstrates state-of-the-art performance across the evaluated datasets.

  • Overall Performance: The method achieves significant improvements over previous approaches, for example, outperforming Majhi et al. [22] by +0.92% and +0.42% on UCF-Crime and Chen et al. [4] by +0.32% and +1.07% on UCF-Crime (official and K-Fold respectively) when using V-Swin as the visual encoder.
  • Long-Short Anomaly Performance: OE-CTST shows strong performance in detecting both long and short anomalies, with notable gains of +3.15% for long anomalies and +0.55% for short anomalies on UCF-C compared to previous methods using V-Swin.
  • Ablation Studies: Experiments confirm the effectiveness of each component, with the CTFA block providing substantial gains over a vanilla transformer, and the OE further boosting performance by effectively incorporating anomaly-aware position information.

Limitations and Future Work

  • Dataset Bias: The authors acknowledge that official test sets for UCF-C and IITB-C can be biased towards certain anomaly categories. The use of K-Fold evaluation helps mitigate this.
  • XD-Violence Annotations: The unavailability of complete temporal annotations for XD-V limits K-Fold evaluation for that dataset.
  • Future Work: While not explicitly detailed, the paper's success with transformers and multi-scale temporal modeling suggests potential for further exploration in more complex scenarios and with larger, more diverse datasets. The flexible nature of the OC-L module also opens avenues for integrating more advanced one-class learning techniques.

Summary updated

BibTeX
@inproceedings{majhi2024oe,
  author = {Snehashis Majhi and Rui Dai and Quan Kong and Lorenzo Garattoni and Gianpiero Francesca and François Brémond},
  title = {{OE-CTST: Outlier-Embedded Cross Temporal Scale Transformer for Weakly-Supervised Video Anomaly Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure