Measuring the Utilization of Public Open Spaces by Deep Learning: a Benchmark Study at the Detroit Riverfront

Peng Sun, Rui Hou, Jerome LynchWACV 2020

TL;DR

This paper introduces a deep learning framework for automatically measuring how people use public open spaces (POS) by analyzing surveillance camera footage. They developed a new dataset of annotated urban park imagery and demonstrated a system that can detect users, recognize their activities, and generate behavioral heatmaps, offering a more efficient and objective alternative to manual observation for urban planning.

What Problem Does This Paper Address?

Understanding how people use public open spaces (POS) like parks and greenways is crucial for effective urban design and planning. Traditionally, this has been done through manual observation, which is time-consuming, labor-intensive, and often yields only qualitative insights. While computer vision offers a promising avenue for automated analysis, existing methods struggle with the complexities of real-world POS environments, such as varying illumination, occlusions, and the need to distinguish between different user activities (e.g., walking vs. cycling). This paper aims to bridge this gap by developing a robust, deep learning-based system for quantitative POS utilization measurement.

Methods

The core of the proposed system is a Mask R-CNN model, a powerful deep learning architecture for object detection and instance segmentation. This model is adapted for user detection and activity recognition within POS. Key methodological aspects include:

  • User Detection and Activity Recognition: Mask R-CNN is employed to not only identify and locate individuals but also to classify their activities. The authors aim to achieve this in a single step by embedding activity recognition within the instance segmentation task.
  • Camera Calibration and 3D Mapping: To translate 2D image detections into real-world 3D locations, the paper utilizes standard pinhole camera models and performs camera calibration. This allows for the estimation of user positions in a 3D coordinate system, enabling the creation of behavioral maps.
  • Behavioral Mapping: The system generates behavioral maps, which are a standard tool in urban design. These maps visualize the distribution of user activities within the POS, providing insights into usage patterns and areas of high/low engagement.
  • Custom Dataset Development: A significant contribution is the creation of a new dataset, "Objects in Public Open Spaces" (OPOS), specifically designed for this task. This dataset features a wide variety of scenarios, including different weather conditions, illumination levels, and diverse user activities.

Datasets

  • OPOS Dataset: This is the primary dataset introduced in the paper. It comprises 7826 annotated images captured from 18 surveillance cameras at the Detroit Riverfront Conservancy (DRFC). The dataset includes 15 object classes, with a focus on people and their activities, and is annotated with instance segmentations. It aims to address the limitations of existing datasets that are often too generic or lack the specific context of POS usage.
  • Pre-training Datasets: For initial model training, the authors leverage established large-scale datasets like ImageNet-1k and COCO_2017_train. This transfer learning approach helps to initialize the model with robust feature representations before fine-tuning on the custom OPOS dataset.

Key Insights and Ideas

  • The Power of Mask R-CNN for POS: The paper demonstrates that Mask R-CNN, with its ability to perform both detection and segmentation, is a suitable architecture for the complex task of identifying and classifying diverse human activities in urban public spaces.
  • Importance of a Specialized Dataset: The creation of the OPOS dataset highlights the necessity of domain-specific data for achieving high performance in specialized computer vision tasks like POS utilization analysis. Generic datasets, while useful for pre-training, often lack the nuanced annotations required for fine-grained activity recognition in these environments.
  • Behavioral Heatmaps as Actionable Insights: The ability to automatically generate behavioral heatmaps provides urban planners and designers with quantitative data on how spaces are actually used. This can inform design decisions, identify underutilized areas, and highlight popular spots, moving beyond subjective observations.
  • Challenges of Small Object Detection: The study reveals that detecting small objects in surveillance footage remains a significant challenge, often leading to localization errors and false negatives. This is attributed to the limited visual information available for such objects.
  • Defining "PeopleOther" is Tricky: The "peopleother" class, intended for severely occluded or truncated individuals, proved difficult to detect accurately. This suggests that clearly defined visual features are crucial for robust recognition, and ambiguity in class definitions can lead to performance degradation.

Key Results

  • Detection Performance: The baseline Mask R-CNN model achieved a mean Average Precision (mAP) of 77.5% for pedestrian detection and 81.6% for cyclist detection on the OPOS dataset.
  • Behavioral Localization Accuracy: The average error for behavioral localization (mapping detected users to real-world coordinates) was within 10 cm.
  • Overall Object Detection: The overall bbox mAP for all objects was 68.4% (67.0% in segmentation).
  • People Super-Category: The mAP for the people super-category was 70.9% (70.5% in segmentation).
  • Specific Class Performance: High AP0.75 values were achieved for common classes like pedestrian (92.6%) and cyclist (95.4%).
  • Small Object Detection: AP for small objects was notably lower, around 63.1%.

Limitations and Future Work

  • Small Object Detection: The paper acknowledges that detecting small objects is a limitation, suggesting that future work could focus on adding more small object annotations to the dataset or exploring specialized detection techniques for such objects.
  • "PeopleOther" Class Ambiguity: The difficulty in detecting the "peopleother" class points to the need for clearer definitions or alternative approaches for handling severely occluded individuals.
  • Future Directions: The authors suggest that their framework can be extended to more complex tasks such as user counting, tracking, and re-identification, which are crucial for comprehensive urban planning studies. They also propose that the developed metrics and error diagnosis methods can spur future research in CV-based POS measurement.

Summary updated

BibTeX
@inproceedings{sun2020measuring,
  author = {Peng Sun and Rui Hou and Jerome Lynch},
  title = {{Measuring the Utilization of Public Open Spaces by Deep Learning: a Benchmark Study at the Detroit Riverfront}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure