TL;DR
This paper introduces "Street Scene," a large and diverse dataset for video anomaly detection, addressing the limitations of existing benchmarks. They also propose new evaluation criteria that better reflect real-world performance and present a novel baseline algorithm that outperforms prior methods on their new dataset.
What Problem Does This Paper Address?
Video anomaly detection aims to automatically identify unusual events in surveillance footage, a crucial task given the impracticality of constant human monitoring. However, current research is hampered by small, unvaried datasets and evaluation metrics that don't accurately reflect an algorithm's practical utility, particularly regarding spatial localization of anomalies. Many existing datasets also suffer from issues like staged anomalies, inconsistent annotations, and low resolution. This paper tackles these issues by providing a more realistic dataset and more robust evaluation methods.
Methods
The authors introduce two variations of a novel baseline algorithm for video anomaly detection. The core idea is to divide video into spatio-temporal "video patches" and learn a set of "exemplars" representing normal activity within each spatial region. During testing, the distance of a new video patch to its nearest exemplar in the same region serves as the anomaly score.
The two variations differ in their feature representation:
- FG Mask Variation: Uses blurred foreground masks derived from a background model.
- Flow-Based Variation: Utilizes optical flow fields computed between consecutive frames.
They also propose new evaluation criteria:
- Track-Based Detection Criterion: Evaluates detection rate against false positives at the track level, considering that an anomaly might span multiple frames.
- Region-Based Detection Criterion: Measures detection rate against false positives at the region level, using Intersection over Union (IOU) for spatial localization.
Datasets
- Street Scene Dataset (New): This is the primary contribution. It features 46 training and 35 testing video sequences from a static USB camera overlooking a two-lane urban street with bike lanes and pedestrian sidewalks. It boasts a large number of frames (203,257 total), a significant number of anomalous events (205 in testing), and a wide variety of "natural" anomalies (17 types), unlike previously staged datasets. The videos are high-resolution (1280x720) and captured at 15 fps.
- Existing Datasets (for comparison/context): UCSD Pedestrian, Subway, CUHK Avenue, UMN, ShanghaiTech, and UCF-Crime are discussed and contrasted with the proposed Street Scene dataset.
Key Insights and Ideas
- The Importance of "Natural" Anomalies: The paper emphasizes that using real-world, unstaged anomalies is crucial for developing algorithms that perform well in practice, as staged anomalies might not capture the full spectrum of unusual events.
- Single-Scene Formulation is Key for Surveillance: The authors argue that for typical surveillance scenarios, learning from a single scene is more effective than pooling data from multiple diverse scenes, as it allows for the detection of location-specific anomalies (e.g., jaywalking in a particular area).
- Spatial Localization Matters: Traditional frame-level evaluation is insufficient. The new track-based and region-based criteria, incorporating IOU, provide a more realistic assessment of an algorithm's ability to pinpoint anomalies spatially.
- FG Masks vs. Optical Flow: The FG mask-based approach shows promise for detecting static anomalies (like loitering) where motion is minimal, while optical flow excels at capturing dynamic anomalies. This suggests that hybrid approaches might be beneficial.
Key Results
- The proposed "Street Scene" dataset is significantly larger and more varied than existing benchmarks for single-scene anomaly detection.
- The authors' novel baseline algorithms (both FG mask and flow-based variations) outperform two previously published methods (dictionary and auto-encoder) on the Street Scene dataset, particularly when evaluated with the new track-based and region-based criteria.
- For the track-based criterion, the flow-based method shows better performance at low false-positive rates.
- For the region-based criterion, the FG mask-based method performs better.
- The auto-encoder and dictionary methods, which performed well on other datasets, struggled significantly on Street Scene, highlighting the dataset's increased complexity and realism.
Limitations and Future Work
- Loitering Anomalies: Static anomalies like loitering are identified as particularly challenging for the flow-based method, as they lack significant motion. The FG-based method shows some ability to detect these.
- Computational Cost: While not explicitly detailed as a limitation, the use of large exemplar sets and spatio-temporal patches suggests potential computational demands.
- Future Work: The authors hope their new dataset and evaluation criteria will spur further research and that researchers will report results on Street Scene. They suggest their nearest-neighbor based algorithms provide a foundation for future work.
Summary updated