Multi-Level Memory-Augmented Appearance-Motion Correspondence Framework for Video Anomaly Detection

Zhiqiang Wu, Xiangyu Huang, Caidan Zhao, Jinhui Yu, Chenxing Gao, Zhiqiang Wu 0001ICME 2023 · DOI

TL;DR

This paper introduces a novel framework for unsupervised video anomaly detection that explicitly models the correspondence between appearance and motion. By leveraging a multi-level memory-augmented approach and a unique "Memory-Guided Suppression Module," the method achieves state-of-the-art performance on challenging benchmarks, effectively distinguishing normal from abnormal events.

What Problem Does This Paper Address?

Video anomaly detection (VAD) aims to identify unusual events in surveillance footage. While many existing methods rely on predicting future frames from past ones, they often struggle because they don't fully capture the interplay between what is happening (appearance) and how it's happening (motion). This can lead to models that "overgeneralize," predicting even anomalous events too well, thus failing to flag them as unusual. The challenge is to build models that understand "normal" behavior by considering both appearance and motion cues simultaneously and robustly, especially in complex scenarios with diverse backgrounds and motions.

Methods

The proposed framework is a multi-level memory-augmented appearance-motion correspondence network. It consists of three main components:

  • Two-Stream Encoder: This part processes both video frames (for appearance) and optical flows (for motion) separately to extract distinct features.
  • Appearance-Motion Semantics Alignment: A novel loss function is introduced to explicitly align the semantic representations of appearance and motion features. Crucially, the framework uses motion features, not appearance features, to predict future frames. This encourages the model to learn a stronger correlation between motion and appearance, and by focusing on foreground motion, it helps mitigate the impact of complex backgrounds.
  • Memory-Guided Suppression Module (MGSM): This is the core innovation. Instead of simply storing normal prototypes in memory, MGSM uses these prototypes to suppress the encoded features. This is implemented within the skip-connections of the autoencoder. By calculating the similarity between encoded features and memory items, MGSM generates a suppressor that is multiplied with the encoded features. This mechanism helps achieve a crucial tradeoff: it allows for good reconstruction of normal data while simultaneously hindering the reconstruction of abnormal data, thus preventing the "overgeneralizing" issue. MGSM operates at multiple levels of the network, leveraging features from different encoding layers.

Datasets

The framework is evaluated on three standard benchmark datasets for video anomaly detection:

  • UCSD Ped2: A dataset of pedestrian traffic with anomalies like abnormal walking or crowds.
  • CUHK Avenue: A dataset of street surveillance with various anomalies such as loitering, running, or object throwing.
  • ShanghaiTech: A more challenging dataset with complex scenes and diverse anomalies.

Key Insights and Ideas

  • Appearance and Motion are Inseparable for True Normality: The paper strongly argues that understanding normality requires jointly modeling appearance and motion. Anomalies often arise from the mismatch between expected appearance and motion, which single-modality approaches miss.
  • Motion-to-Frame Prediction is Key: Using motion features to predict future frames, rather than appearance features, is a clever way to enforce the appearance-motion correlation and reduce background noise.
  • Suppression, Not Just Memorization: The MGSM's approach of suppressing features based on memory prototypes is more effective than traditional memory-based methods that solely rely on memorization for prediction. This suppression directly targets the problematic skip-connections that can lead to overgeneralization.
  • Adaptive Reconstruction is Crucial: The MGSM's ability to adaptively balance reconstruction of normal vs. abnormal data is vital for robust anomaly detection, especially in complex environments.

Key Results

The proposed framework achieves state-of-the-art performance on all three benchmark datasets:

  • UCSD Ped2: AUC of 99.6%
  • CUHK Avenue: AUC of 93.8%
  • ShanghaiTech: AUC of 76.3%

These results represent significant improvements over existing methods, particularly on the challenging ShanghaiTech dataset. Ablation studies confirm the effectiveness of each proposed component, with the MGSM showing a substantial boost in AUC performance. The paper also demonstrates that the MGSM is robust to the number of memory items, unlike some previous memory-based approaches.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, memory-augmented networks and multi-stream architectures can be computationally intensive.
  • Generalizability to Extremely Rare Anomalies: The assumption that anomalies are "unexpected" might still be challenged by extremely rare or novel anomaly types not captured by the "normal" patterns learned.
  • Future Work: The authors suggest further exploration of the MGSM's adaptability to different complex scenes and potentially extending the framework to other video understanding tasks. The paper also hints at the need for more sophisticated memory management and retrieval mechanisms for even greater robustness.

Summary updated

BibTeX
@inproceedings{wu2023multi,
  author = {Zhiqiang Wu and Xiangyu Huang and Caidan Zhao and Jinhui Yu and Chenxing Gao and Zhiqiang Wu},
  title = {{Multi-Level Memory-Augmented Appearance-Motion Correspondence Framework for Video Anomaly Detection}},
  booktitle = {ICME},
  year = {2023},
  doi = {10.1109/icme55011.2023.00462},
}
Figure