Attentioned Convolutional LSTM InpaintingNetwork for Anomaly Detection in Videos

Itamar Ben-Ari, Ravid Shwartz-ZivarXiv 2018 · OpenReview

TL;DR

This paper introduces a novel semi-supervised approach for detecting anomalies in videos by adapting the Video Pixel Network (VPN). Their method enhances VPN with a convolutional attention mechanism and a frame inpainting task, allowing it to reconstruct masked video frames and identify anomalies based on reconstruction errors.

What Problem Does This Paper Address?

Real-time anomaly detection in videos is crucial for applications like robot patrolling and manufacturing quality control. However, the complexity and variability of video data, coupled with high computational demands, make this a challenging task. Existing methods often rely on supervised learning or struggle with capturing both temporal and spatial dynamics effectively. This paper aims to develop an unsupervised method that can accurately detect anomalies by learning normal video patterns and flagging deviations.

Methods

The proposed model builds upon the Video Pixel Network (VPN), a generative model that predicts future video frames. The authors introduce two key modifications:

  1. Convolutional Attention Mechanism: Instead of standard attention mechanisms that might overlook local image structures, they employ a convolutional attention mechanism. This uses input-dependent filter weights generated by a small meta-network. This allows the model to dynamically focus on relevant parts of the video context while preserving spatial information.
  2. Frame Inpainting Task: The original VPN uses a Pixel-CNN decoder for frame prediction. This paper modifies the decoder to perform frame inpainting. Instead of predicting a full frame, it takes a partially masked version of the frame as input and reconstructs it. The idea is that the model, trained on anomaly-free videos, will learn to reconstruct normal patterns. When an anomaly is present, the reconstruction error will be high, signaling an anomaly. This masked input also enables parallel processing during inference.

The loss function used is the negative log-likelihood of the pixel values given the predicted distribution, serving as an anomaly measure. Low log-likelihood values indicate a higher chance of an anomaly.

Datasets

  • Moving MNIST Dataset: The authors use a modified version of the Moving MNIST dataset. This dataset consists of sequences of moving digits within a frame. For training, sequences are generated on-the-fly. For testing, they use both normal and corrupted sequences. Corruptions include replacing the last frame with the first (temporal anomaly) and painting a black square on a digit (spatial anomaly).

Key Insights and Ideas

  • Attention for Context: The convolutional attention mechanism is crucial for effectively leveraging the spatial structure of video frames, leading to better context understanding than global attention.
  • Inpainting as Anomaly Detection: Framing anomaly detection as a frame inpainting task is an elegant way to leverage reconstruction error. By masking most of the frame, the model is forced to rely on learned temporal and spatial patterns from the surrounding unmasked pixels and previous frames.
  • Unsupervised Learning Power: The model demonstrates the effectiveness of unsupervised learning for anomaly detection, learning normal video behavior without explicit labels for anomalies.
  • Parallel Inference: The masked frame input strategy allows for parallel prediction of pixels during inference, contributing to real-time anomaly detection capabilities.

Key Results

The paper reports quantitative results using the Equal Error Rate (EER) metric. Their proposed model achieves an EER of 87%, outperforming baseline models like the original VPN (82.1%) and Conv-LSTM (70.3%). Notably, their model with both convolutional attention and masked frame inpainting performs best. Ablation studies show that removing either the masked convolutions or the convolutional attention mechanism leads to a decrease in performance (EER of 84.6% and 85.7%, respectively), highlighting the importance of each component. Qualitative results (Figure 1) show the model successfully identifying both spatial and temporal anomalies by producing high cross-entropy loss maps in corrupted regions.

Limitations and Future Work

  • Dataset Specificity: The model was evaluated on the Moving MNIST dataset, which is relatively simple. Its performance on more complex, real-world video datasets needs further investigation.
  • Computational Cost: While the masked frame approach enables parallel inference, the overall computational cost of the attention mechanism and convolutional layers might still be a consideration for extremely resource-constrained environments.
  • Future Work: The authors suggest that this approach could be a component in applications requiring "visual common sense," implying potential for integration into more complex AI systems. Further exploration on diverse datasets and real-world scenarios would be a natural next step.

Summary updated

Abstract

We propose a semi-supervised model for detecting anomalies in videos inspiredby the Video Pixel Network [van den Oord et al., 2016]. VPN is a probabilisticgenerative model based on a deep neural network that estimates the discrete jointdistribution of raw pixels in video frames. Our model extends the Convolutional-LSTM video encoder part of the VPN with a novel convolutional based attentionmechanism. We also modify the Pixel-CNN decoder part of the VPN to a frameinpainting task where a partially masked version of the frame to predict is given asinput. The frame reconstruction error is used as an anomaly indicator. We test ourmodel on a modified version of the moving mnist dataset [Srivastava et al., 2015]. Our model is shown to be effective in detecting anomalies in videos. This approachcould be a component in applications requiring visual common sense.
BibTeX
@misc{benari2018attentioned,
  author = {Itamar Ben-Ari and Ravid Shwartz-Ziv},
  title = {{Attentioned Convolutional LSTM InpaintingNetwork for Anomaly Detection in Videos}},
  year = {2018},
  eprint = {1811.10228},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.10228},
}
Figure