A Revisit of Sparse Coding Based Anomaly Detection in Stacked RNN Framework

Weixin Luo, Wen Liu, Shenghua GaoICCV 2017 · OpenReview

Abstract

Motivated by the capability of sparse coding based anomaly detection, we propose a Temporally-coherent Sparse Coding (TSC) where we enforce similar neighbouring frames be encoded with similar reconstruction coefficients. Then we map the TSC with a special type of stacked Recurrent Neural Network (sRNN). By taking advantage of sRNN in learning all parameters simultaneously, the nontrivial hyper-parameter selection to TSC can be avoided, meanwhile with a shallow sRNN, the reconstruction coefficients can be inferred within a forward pass, which reduces the computational cost for learning sparse coefficients. The contributions of this paper are two-fold: i) We propose a TSC, which can be mapped to a sRNN which facilitates the parameter optimization and accelerates the anomaly prediction. ii) We build a very large dataset which is even larger than the summation of all existing dataset for anomaly detection in terms of both the volume of data and the diversity of scenes. Extensive experiments on both a toy dataset and real datasets demonstrate that our TSC based and sRNN based method consistently outperform existing methods, which validates the effectiveness of our method.

Introduction: The Challenge of Video Anomaly Detection

Anomaly detection in video—identifying rare events like a vehicle on a sidewalk or a sudden crowd disturbance—is a fundamental challenge in computer vision. The primary difficulty lies in the unbounded nature of "anomalies"; it is impossible to train a model on every conceivable irregular event. Traditionally, this was addressed via Dictionary Learning and Sparse Coding. Researchers would learn a dictionary of "normal" patterns; any frame that could not be reconstructed with low error using this dictionary was flagged as anomalous.

However, classical sparse coding suffers from two significant bottlenecks:

  1. Computational Latency: Inferring sparse coefficients is an iterative optimization process, making real-time application difficult.
  2. Hyper-parameter Sensitivity: Tuning parameters like the sparsity penalty often requires manual, scene-specific adjustment.

Methodology: Mapping Sparse Coding to Stacked RNNs

To overcome these limitations, this work introduces Temporally-coherent Sparse Coding (TSC) and its neural network implementation, the stacked Recurrent Neural Network (sRNN).

Temporal Coherence

Unlike standard methods that treat frames in isolation, TSC enforces a temporal constraint: neighboring frames with similar appearances should yield similar sparse representations. This ensures a smoother, more reliable "regularity score" over time.

The sRNN Framework

The core innovation is the mathematical mapping of the Iterative Soft-Thresholding Algorithm (ISTA)—the standard solver for sparse coding—into the architecture of a stacked RNN. By "unrolling" the optimization steps into network layers, the dictionary ($A$) and sparsity weights ($\lambda$) can be learned end-to-end via backpropagation.

Comparison between a vanilla stacked RNN and the proposed sRNN architecture designed to mimic the TSC optimization process.

In the testing phase, the model bypasses slow iterative loops. Instead, it performs a single forward pass through the sRNN to calculate the reconstruction error, enabling lightning-fast inference.

Benchmarking and the ShanghaiTech Dataset

The proposed method was evaluated against standard benchmarks including CUHK Avenue, UCSD Ped1/Ped2, and the Subway datasets.

Visual examples of normal and anomalous events across various standard datasets used for evaluation.

Recognizing that existing datasets lacked scale and scene diversity, the authors introduced the ShanghaiTech Campus Dataset. This new benchmark contains over 270,000 training frames across 13 different scenes, featuring complex anomalies like brawling and chasing that were previously underrepresented in the literature.

Key Results and Insights

  • Superior Accuracy: The sRNN framework consistently outperformed state-of-the-art methods. On the UCSD Ped2 dataset, it achieved a frame-level AUC of 92.21%, a substantial improvement over the 81.1% achieved by previous deep learning benchmarks like Conv-AE.
  • Real-Time Performance: The sRNN is approximately 10 times faster than iterative TSC, processing frames at roughly 50 fps (0.02 seconds per frame).
  • The "Shallow" Advantage: Experiments revealed that a relatively shallow sRNN ($K=3$) often outperformed deeper versions. This suggests that for anomaly detection, a robust "rough" sparse approximation is more effective than a mathematically perfect one, which may overfit to noise.
  • Feature Universality: Utilizing high-level spatial features from a pre-trained VGG network proved more effective for dictionary learning than traditional hand-crafted features like HOG or HOF.

Conclusion and Future Directions

By bridging classical signal processing with deep learning, this work demonstrates that unrolling optimization algorithms into neural architectures provides both interpretability and efficiency. While the sRNN handles spatial context exceptionally well, future work may explore more direct integration of motion features or advanced gating mechanisms to further enhance temporal modeling.

Summary updated

Abstract

Motivated by the capability of sparse coding based anomaly detection, we propose a Temporally-coherent Sparse Coding (TSC) where we enforce similar neighbouring frames be encoded with similar reconstruction coefficients. Then we map the TSC with a special type of stacked Recurrent Neural Network (sRNN). By taking advantage of sRNN in learning all parameters simultaneously, the nontrivial hyper-parameter selection to TSC can be avoided, meanwhile with a shallow sRNN, the reconstruction coefficients can be inferred within a forward pass, which reduces the computational cost for learning sparse coefficients. The contributions of this paper are two-fold: i) We propose a TSC, which can be mapped to a sRNN which facilitates the parameter optimization and accelerates the anomaly prediction. ii) We build a very large dataset which is even larger than the summation of all existing dataset for anomaly detection in terms of both the volume of data and the diversity of scenes. Extensive experiments on both a toy dataset and real datasets demonstrate that our TSC based and sRNN based method consistently outperform existing methods, which validates the effectiveness of our method.
BibTeX
@inproceedings{luo2017revisit,
  author = {Weixin Luo and Wen Liu and Shenghua Gao},
  title = {{A Revisit of Sparse Coding Based Anomaly Detection in Stacked RNN Framework}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  url = {https://openreview.net/forum?id=B0uUWaoQ6bP},
}
Figure