TL;DR
This paper introduces a novel approach to video anomaly detection by training a Siamese Convolutional Neural Network (CNN) to learn a distance function between video patches. This learned distance function, trained on diverse datasets, allows for efficient anomaly localization by measuring how dissimilar a test video patch is to normal exemplars, achieving state-of-the-art results on benchmark datasets.
What Problem Does This Paper Address?
Video anomaly detection aims to identify unusual activities in surveillance footage. A key challenge is that "normal" behavior is highly scene-dependent, making it difficult to build a universal model. Typically, anomaly detection relies on having examples of normal activity to train a model. However, collecting anomalous video data for training is often impractical. This paper tackles this by reframing the problem: instead of directly classifying patches as normal or anomalous, it focuses on learning a robust distance metric that can distinguish between normal and abnormal video content, even when trained on data from different sources than the target video.
Methods
The core of the proposed method is a Siamese CNN. This network architecture consists of two identical "twin" convolutional tails that process pairs of video patches. These tails share weights, ensuring that similar inputs produce similar intermediate representations. After processing, the feature vectors from the two tails are element-wise subtracted and fed into a decision network that outputs a probability of the pair being similar or dissimilar.
The training process is crucial:
- Generating Training Pairs: The Siamese network is trained on pairs of video patches labeled as "similar" (0) or "dissimilar" (1). These pairs are generated by leveraging existing video anomaly detection datasets. For similar pairs, normal video patches from a test set are matched with their closest counterparts in the training set. For dissimilar pairs, anomalous video patches from the test set are identified as being distinct from all training patches.
- Distance Function Learning: The network is trained using a cross-entropy loss function. The goal is to learn a distance function that accurately reflects the similarity between video patches. The authors emphasize that the network is trained on a diverse set of source datasets (UCSD Ped2, ShanghaiTech, CUHK Avenue) and is not trained on the target dataset itself, promoting generalization.
- Anomaly Scoring: At test time, the learned Siamese network is used to measure the distance between video patches from the target video and a set of "exemplars" representing normal activity for that specific region. An anomaly score is then computed as the minimum distance to any exemplar.
Datasets
The paper uses the following datasets:
- Training Source Datasets: UCSD Ped2, ShanghaiTech, and CUHK Avenue are used to train the Siamese network.
- Evaluation Datasets: Experiments are conducted on UCSD Ped1, UCSD Ped2, and CUHK Avenue.
The authors note that the training and testing partitions for each dataset are kept separate, and the source datasets used for training are disjoint from the target dataset.
Key Insights and Ideas
- Reframing the Problem: The most significant insight is shifting the focus from direct anomaly classification (which requires anomalous training data) to learning a robust distance metric. This makes the problem tractable even with only normal training data.
- Siamese Network for Distance Learning: Using a Siamese network with shared weights is an effective way to learn a comparative representation and a distance function between video patches.
- Cross-Dataset Training for Generalization: Training the Siamese network on a diverse set of source datasets, independent of the target dataset, allows the learned distance function to generalize better to unseen scenarios.
- Exemplar-Based Anomaly Detection: The learned distance function is then seamlessly integrated into a simple exemplar-based nearest neighbor approach, making the anomaly detection process interpretable.
Key Results
The proposed method achieves state-of-the-art or comparable performance on benchmark datasets:
- Pixel-Level AUC: Achieved approximately 4% improvement on UCSD Ped2 pixel-level AUC.
- Frame-Level EER: Achieved approximately 6% improvement on CUHK Avenue frame EER.
- New Criteria: Outperforms previous methods on region-based and track-based anomaly detection criteria, detecting more anomalous events with fewer false positives.
- Qualitative Findings: The method is effective in detecting a wide variety of anomalies. However, it struggles with very small-scale anomalies and loitering anomalies, which are noted as limitations.
Limitations and Future Work
- Limitations:
- The method finds it difficult to detect anomalies at very small scales, particularly in the UCSD Ped1 dataset.
- Loitering anomalies are not well-detected, partly due to the "motion check" that can ignore patches with little motion.
- The brute-force nearest neighbor search used in experiments can be computationally intensive.
- Future Work:
- Developing more sophisticated background models to handle stationary objects better.
- Replacing the brute-force nearest neighbor search with faster approximate methods.
- Investigating training a single Siamese network that works well across many different target datasets, rather than training a separate network for each target.
Summary updated