Dynamic Erasing Network Based on Multi-Scale Temporal Features for Weakly Supervised Video Anomaly Detection

Chen Zhang, Guorong Li, Yuankai Qi, Hanhua Ye, Laiyun Qing, Ming-Hsuan Yang, Qingming HuangarXiv 2023 · DOI · arXiv

Abstract

Weakly supervised video anomaly detection (WSVAD) is a critical task that involves identifying abnormal events in videos using only video-level labels. Current state-of-the-art methods often suffer from two major issues: "fixed-scale blindness," where models fail to adapt to varying event durations, and "peak bias," where models focus only on the most salient moments of an anomaly. We introduce the Dynamic Erasing Network (DE-Net), which addresses these challenges through Multi-Scale Temporal Modeling (MSTM) and a novel Dynamic Erasing (DE) strategy. MSTM captures features across multiple temporal resolutions, while the DE strategy forces the model to discover subtle anomalous cues by iteratively erasing prominent segments during training. DE-Net achieves new state-of-the-art results on the XD-Violence, UCF-Crime, and TAD datasets.

Introduction

Video anomaly detection in real-world scenarios is complex due to the diversity of abnormal events. In the weakly supervised setting, models are trained without frame-level annotations, often leading them to identify only the most obvious "peaks" of an anomaly while missing the full context or the event's start and end points. This is known as the "completeness issue."

Figure 1: Illustration of the completeness issue. Traditional models (b1) produce spiky scores for long anomalies (b), whereas DE-Net (b2) captures the full duration with a plateau-like response.

DE-Net is designed to overcome these limitations by ensuring the model "looks harder" at the entire video sequence, rather than stopping at the first obvious visual cue it finds.

Methodology

The DE-Net framework is built upon two core technical innovations: multi-scale feature extraction and an adversarial erasing mechanism.

Multi-Scale Temporal Modeling (MSTM)

To account for the fact that anomalies like explosions are brief while events like fights can be prolonged, the MSTM module utilizes multiple 1D convolutional layers with varying strides and kernel sizes. This allows the network to process the video at different temporal resolutions simultaneously. These multi-scale features are then integrated using a Transformer encoder to capture global temporal context.

Dynamic Erasing (DE) Strategy

The Dynamic Erasing strategy acts as a regularizer during training. The model calculates a "completeness score" by comparing the similarity between the most abnormal and most normal segments. If an anomaly is deemed incomplete, the model erases the most prominent abnormal segments by zeroing out their features. This forces the classifier to identify secondary, more subtle cues it might have otherwise ignored, leading to a more robust and complete representation of the anomalous event.

Figure 2: The DE-Net architecture, highlighting the Multi-Scale Temporal Modeling (MSTM) and Dynamic Erasing (DE) modules used during training.

Experiments and Results

DE-Net was rigorously tested on three major benchmarks, consistently outperforming existing methods:

  • XD-Violence: The model reached 81.66% AP using visual features (I3D) and 83.13% AP when combined with audio features (VGGish), outperforming the previous best (CU-Net) by nearly 3%.
  • UCF-Crime: DE-Net achieved 86.33% AUC, maintaining top-tier performance on fixed-camera surveillance data.
  • TAD (Traffic Anomaly Dataset): The model achieved 93.10% AUC, a 1.44% improvement over previous state-of-the-art results.

Ablation studies demonstrated that the multi-scale approach alone provided a 1.44% boost in Average Precision, while the dynamic erasing mechanism was crucial for transforming "spiky" anomaly scores into accurate plateaus that cover the entire duration of an incident.

Conclusion

By combining multi-scale temporal modeling with a dynamic erasing strategy, DE-Net effectively solves the completeness issue in weakly supervised video anomaly detection. The model's ability to identify both obvious and subtle anomalous cues makes it a robust solution for complex, real-world video analysis. Future work will focus on refining multi-modal fusion and exploring adaptive erasing thresholds for different anomaly categories.

Summary updated

Abstract

The goal of weakly supervised video anomaly detection is to learn a detection model using only video-level labeled data. However, prior studies typically divide videos into fixed-length segments without considering the complexity or duration of anomalies. Moreover, these studies usually just detect the most abnormal segments, potentially overlooking the completeness of anomalies. To address these limitations, we propose a Dynamic Erasing Network (DE-Net) for weakly supervised video anomaly detection, which learns multi-scale temporal features. Specifically, to handle duration variations of abnormal events, we first propose a multi-scale temporal modeling module, capable of extracting features from segments of varying lengths and capturing both local and global visual information across different temporal scales. Then, we design a dynamic erasing strategy, which dynamically assesses the completeness of the detected anomalies and erases prominent abnormal segments in order to encourage the model to discover gentle abnormal segments in a video. The proposed method obtains favorable performance compared to several state-of-the-art approaches on three datasets: XD-Violence, TAD, and UCF-Crime. Code will be made available at https://github.com/ArielZc/DE-Net.
BibTeX
@misc{zhang2023dynamic,
  author = {Chen Zhang and Guorong Li and Yuankai Qi and Hanhua Ye and Laiyun Qing and Ming-Hsuan Yang and Qingming Huang},
  title = {{Dynamic Erasing Network Based on Multi-Scale Temporal Features for Weakly Supervised Video Anomaly Detection}},
  year = {2023},
  eprint = {2312.01764},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.01764},
}
Figure