Introduction
Monitoring surveillance cameras is a "needle in a haystack" problem, notoriously difficult for humans due to fatigue and the sheer volume of data. While automated systems exist, they often rely on complex architectures like Generative Adversarial Networks (GANs) or specialized spatio-temporal models that are difficult to train and tune. This research explores whether we can leverage "off-the-shelf" knowledge from models trained on general images (like ImageNet) to detect anomalies in surveillance footage without domain-specific training. The authors investigate the effectiveness of transfer learning as a "plug-and-play" feature extractor for detecting unusual behavior in video, providing a baseline for modern architectures.
Methodology
The researchers proposed a three-step pipeline to evaluate four state-of-the-art architectures: VGG-16, ResNet-50, Xception, and DenseNet-121.
- Feature Extraction: Instead of processing whole frames, video frames were decomposed into $32 \times 32$ patches. These patches were passed through the convolutional layers of the pre-trained CNNs. By utilizing only the convolutional part, the researchers avoided the fixed-input-size constraints typically imposed by fully connected layers.
- Data Preparation: This stage proved critical to performance. Four normalization techniques (z-score, 0-1, L1, and L2) were tested. Additionally, Incremental PCA (IPCA) was used to reduce massive feature vectors down to 50 or 100 dimensions. IPCA was selected for its memory efficiency, allowing it to handle large datasets that exceed available RAM.
- Anomaly Detection: A simple 1-Nearest Neighbor (1-NN) approach was employed. If a patch in a test video was mathematically distant from any patch seen in the training set (which contains only normal behavior), it was flagged as an anomaly.
Datasets and Experimental Setup
The study utilized two classic benchmarks from the University of California, San Diego:
- UCSD Ped1: Contains 34 training and 36 test videos of a pedestrian walkway with significant perspective distortion, where people appear smaller as they move away from the camera.
- UCSD Ped2: Contains 16 training and 12 test videos of a different scene with less perspective change.
Anomalies in these datasets typically include non-pedestrian entities like bikers, skaters, and small carts on pedestrian paths.
Key Results and Insights
The performance was evaluated using the Area Under the ROC Curve (AUC) and Equal Error Rate (EER).
- Normalization Impact: The choice of normalization was the most striking factor in model performance. For instance, ResNet-50 performed best with z-score normalization, while DenseNet-121 required 0-1 scaling. Selecting the incorrect technique could significantly degrade accuracy.
- Appearance vs. Motion: By intentionally ignoring motion, the authors found that many anomalies (e.g., a car on a sidewalk) are detectable based solely on shape and texture. On the Ped2 dataset, the Xception model achieved a competitive AUC of 88.93%, comparable to specialized state-of-the-art methods of the time.
- The Perspective Challenge: Performance dropped significantly on the Ped1 dataset, with the best AUC reaching only 64.06% (VGG-16). This highlights the limitation of simple patch-based models in scenes with high perspective variance, where the pixel-level appearance of an object changes drastically based on its distance from the camera.
Conclusion and Future Work
While pre-trained CNNs serve as effective "appearance-only" feature extractors, the omission of temporal data prevents the detection of behavioral anomalies, such as loitering or walking in the wrong direction. Future research paths include integrating motion features (like optical flow), investigating the descriptive power of CNNs on grayscale vs. RGB data, and exploring alternative one-class classification algorithms like Isolation Forests to better handle high-dimensional CNN features.
Summary updated