A Large Scale Urban Surveillance Video Dataset for Multiple-Object Tracking and Behavior Analysis

Guojun Yin, Bin Liu, Huihui Zhu, Tao Gong, Nenghai YuarXiv 2019 · arXiv

Abstract

Researchers from the University of Science and Technology of China have introduced the Urban Surveillance Video Dataset (USVD), a comprehensive benchmark designed to advance multi-object tracking (MOT) and behavior analysis. Comprising over 3.7 million bounding boxes across 211,706 frames, USVD addresses the limitations of existing "clean" datasets by capturing the high congestion, frequent occlusions, and diverse viewpoints characteristic of real-world urban surveillance.

Introduction

While deep learning has significantly advanced computer vision, standard benchmarks like KITTI or the MOT Challenge often lack the complexity of 24/7 urban environments. In real-world city surveillance, cameras frequently encounter extreme congestion where dozens of pedestrians, cyclists, and vehicles overlap. Current algorithms often struggle when transitioning from sparse laboratory data to these dense, noisy environments. USVD fills this gap by providing a large-scale, realistic dataset designed to stress-test the next generation of urban AI.

Methods and Data Curation

The authors curated USVD by collecting nearly 4,000 hours of raw video from 54 surveillance cameras across diverse outdoor scenarios, including crossroads, school gates, and parks. From this pool, 16 representative scenes were selected for manual annotation.

The USVD dataset captures a wide variety of urban scenarios, including crossroads, school gates, and public parks.

The annotation process followed strict consistency rules. Seven distinct classes were identified: pedestrian, riding (person + vehicle), car, van, bus, tricycle, and truck. Notably, a person on a bicycle or scooter is treated as a single "riding" unit, reflecting how urban planners and security systems typically view traffic flow. To maintain accuracy in crowded scenes, targets smaller than 60 pixels were handled with specific protocols.

The USVD Dataset

USVD represents a significant leap in scale and density compared to existing benchmarks. It contains 32 sequences and approximately 7,100 unique trajectories. With over 3.7 million bounding boxes, it is more than ten times the size of the popular MOT Challenge 2017. The dataset's density is a key feature, averaging 18-20 objects per frame. This high density ensures that models must account for frequent "mutual occlusion," where objects block one another, rather than just simple motion patterns.

Key Insights: Scale Variation and Congestion

A primary challenge identified in USVD is extreme scale variation. As objects approach or move away from the camera, their size in pixels changes dramatically. The authors found that over 32% of targets undergo "huge" scale changes, which is a frequent cause of tracking failure in standard models.

Distribution of pedestrian heights and car widths in USVD, highlighting the high percentage of small-scale targets that challenge tracking algorithms.

Experimental Results

The authors established baselines using standard detection and tracking models:

  • Detection: The Single Shot MultiBox Detector (SSD) achieved a mean Average Precision (mAP) of 0.6656, slightly outperforming Faster R-CNN and demonstrating robustness for real-time urban applications.
  • Tracking: The TC_ODAL algorithm reached the highest Multiple Object Tracking Accuracy (MOTA) at 41.05%. Interestingly, the simpler IOU tracker performed nearly as well (40.94%), suggesting that spatial overlap is a powerful cue in high-density scenes.
  • Anomaly Detection: Using the Histogram of Optical Flow (HMOF) method, the researchers achieved an Area Under Curve (AUC) of 0.883 for identifying anomalous behaviors like running in restricted zones.

Conclusion and Future Work

USVD provides a rigorous new standard for urban surveillance AI. While currently restricted by licensing, its scale and complexity offer a unique testing ground for scale-invariant and occlusion-robust algorithms. Future work includes annotating "group" dynamics (people walking together) and expanding the dataset to include varied weather conditions such as rain, snow, and nighttime footage to further challenge AI robustness.

Summary updated

Abstract

Multiple-object tracking and behavior analysis have been the essential parts of surveillance video analysis for public security and urban management. With billions of surveillance video captured all over the world, multiple-object tracking and behavior analysis by manual labor are cumbersome and cost expensive. Due to the rapid development of deep learning algorithms in recent years, automatic object tracking and behavior analysis put forward an urgent demand on a large scale well-annotated surveillance video dataset that can reflect the diverse, congested, and complicated scenarios in real applications. This paper introduces an urban surveillance video dataset (USVD) which is by far the largest and most comprehensive. The dataset consists of 16 scenes captured in 7 typical outdoor scenarios: street, crossroads, hospital entrance, school gate, park, pedestrian mall, and public square. Over 200k video frames are annotated carefully, resulting in more than 3:7 million object bounding boxes and about 7:1 thousand trajectories. We further use this dataset to evaluate the performance of typical algorithms for multiple-object tracking and anomaly behavior analysis and explore the robustness of these methods in urban congested scenarios.
BibTeX
@misc{yin2019large,
  author = {Guojun Yin and Bin Liu and Huihui Zhu and Tao Gong and Nenghai Yu},
  title = {{A Large Scale Urban Surveillance Video Dataset for Multiple-Object Tracking and Behavior Analysis}},
  year = {2019},
  eprint = {1904.11784},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.11784},
}
Figure