Dual-Stream Attention-Enhanced Memory Networks for Video Anomaly Detection

Wei Gao, Xiaoyin Wang, Ye Wang, Xiaochuan JingSensors 2025 · DOI

TL;DR

This paper introduces a novel "Dual-Stream Attention-Enhanced Memory Network" (DS-AEMN) for weakly supervised video anomaly detection. By combining a hierarchical temporal encoder with a dual-memory system that learns both normal and abnormal patterns, the method significantly improves accuracy and reduces false alarms in identifying unusual events in videos, achieving state-of-the-art results.

What Problem Does This Paper Address?

Video anomaly detection (VAD) is crucial for intelligent surveillance, but identifying unusual events in long, unedited videos is challenging. Current methods, especially weakly supervised ones (which only use video-level labels), struggle with:

  • Complex Temporal Dependencies: Difficulty in modeling how events unfold over time, both short-term and long-term.
  • Indistinct Feature Boundaries: Similar visual characteristics between normal and abnormal events make it hard to differentiate them.
  • High False Alarm Rates: Models often misinterpret normal but unusual occurrences (like sudden lighting changes) or background noise as anomalies.

These issues lead to models that are either inaccurate or generate too many false positives, limiting their real-world applicability.

Methods

The proposed method, DS-AEMN, tackles these challenges with a multi-component architecture:

  • Hierarchical Multi-Scale Temporal Encoder (HMTE): This module uses stacked dilated convolutions to capture temporal information at various scales, from local movements to long-range dependencies. It incorporates residual connections to prevent feature degradation in deep networks.
  • Position-Aware Global Relation Network (PGRN): To complement HMTE's local focus, PGRN uses a modified self-attention mechanism to capture global relationships between non-adjacent keyframes, considering both content and temporal proximity.
  • Dual-Stream Attention-Enhanced Memory Network (DS-AEMN): This is the core innovation.
  • Dual-Stream Attention: A bidirectional spatial and channel attention mechanism refines features by focusing on critical spatial regions ("where") and informative feature channels ("what"), effectively filtering out background noise.
  • Dual Memory Banks: Instead of a single memory for normal patterns, DS-AEMN uses two separate, trainable memory banks: one for "normal" prototypes and one for "abnormal" prototypes. This allows the model to learn distinct representations for both, leading to better discrimination.
  • Attention-Guided Prototype Matching: The refined features from the attention module are used as queries to match against the learned prototypes in the memory banks, enabling precise anomaly differentiation.

The entire system is trained using a multi-objective composite loss function that balances classification, memory compactness, and feature separation.

Datasets

The proposed method was evaluated on two widely used public datasets for video anomaly detection:

  • UCF-Crime: Contains 13 categories of anomalous incidents from surveillance videos. It's known for having subtle anomalies and significant background dynamics.
  • XD-Violence: A large and heterogeneous dataset with diverse anomalies and challenging video-level labels, making it suitable for testing robustness and generalization.

Key Insights and Ideas

  • Dual Memory for Dual Nature: The key insight is that explicitly learning both normal and abnormal prototypes is more effective than just learning what's normal and detecting deviations. This dual approach directly addresses the indistinct boundaries between normal and abnormal events.
  • Attention as a Pre-Filter: Using attention mechanisms before querying memory banks acts as a crucial denoising and focusing step, significantly improving the model's ability to ignore irrelevant background information and concentrate on salient features.
  • Hierarchical Temporal Modeling: Capturing temporal dynamics at multiple scales (local and global) is essential for understanding how events unfold, preventing information loss in deep networks.
  • "Attention First, Then Memory Matching": This paradigm effectively combines feature refinement with learned knowledge, leading to robust and interpretable anomaly detection.

Key Results

  • State-of-the-Art Performance: The DS-AEMN achieved state-of-the-art results on both datasets:
  • UCF-Crime: 87.43% AUC (Area Under the ROC Curve).
  • XD-Violence: 85.51% AP (Average Precision).
  • Reduced False Alarms: The model demonstrated a significantly lower False Alarm Rate (FAR) compared to other methods, particularly in windowed evaluations (5.21% vs. 11.60% and 21.40% for comparison methods).
  • Computational Efficiency: The model is highly efficient, requiring only 0.01G parameters and achieving a fast test time of 0.03s, making it suitable for real-world deployment.
  • Fine-Grained Discrimination: The model showed strong performance in differentiating visually similar anomalies (e.g., "stealing," "shoplifting," "robbery"), with high recall for the "stealing" class (98.4%).

Limitations and Future Work

  • Cross-Domain Generalization: The authors acknowledge that enhancing cross-domain generalization is a future direction. They propose investigating adaptive, unsupervised updating mechanisms for memory prototypes to reduce reliance on pseudo-labels.
  • Multimodal Integration: Currently, the model uses only RGB inputs. Future work will explore developing cross-modal attention mechanisms that incorporate other modalities like optical flow or audio data to improve detection in scenarios with weak visual cues.
  • Fine-grained Analysis: While the paper presents qualitative analysis and confusion matrices, a more detailed quantitative analysis of anomaly localization accuracy, especially for varying durations, is suggested for future work.
  • Hyperparameter Sensitivity: While the paper conducts sensitivity analysis, the performance is somewhat contingent on the composite loss function's hyperparameters (λ values). The chosen values are presented as robust, but further exploration might be beneficial.

Summary updated

Abstract

Weakly supervised video anomaly detection (WSVAD) aims to identify unusual events using only video-level labels. However, current methods face several key challenges, including ineffective modelling of complex temporal dependencies, indistinct feature boundaries between visually similar normal and abnormal events, and high false alarm rates caused by an inability to distinguish salient events from complex background noise. This paper proposes a novel method that systematically enhances feature representation and discrimination to address these challenges. The proposed method first builds robust temporal representations by employing a hierarchical multi-scale temporal encoder and a position-aware global relation network to capture both local and long-range dependencies. The core of this method is the dual-stream attention-enhanced memory network, which achieves precise discrimination by learning distinct normal and abnormal patterns via dual memory banks, while utilising bidirectional spatial attention to mitigate background noise and focus on salient events before memory querying. The models underwent a comprehensive evaluation utilising solely RGB features on two demanding public datasets, UCF-Crime and XD-Violence. The experimental findings indicate that the proposed method attains state-of-the-art performance, achieving 87.43% AUC on UCF-Crime and 85.51% AP on XD-Violence. This result demonstrates that the proposed "attention-guided prototype matching" paradigm effectively resolves the aforementioned challenges, enabling robust and precise anomaly detection.
BibTeX
@inproceedings{gao2025dual,
  author = {Wei Gao and Xiaoyin Wang and Ye Wang and Xiaochuan Jing},
  title = {{Dual-Stream Attention-Enhanced Memory Networks for Video Anomaly Detection}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25175496},
}
Figure