4D+ City Sidewalk: Integrating Pedestrian View into Sidewalk Spaces to Support User-Centric Urban Spatial Perception

Jinjing Zhao, Yunfan Chen, Yancheng Li, Haotian Xu, Jingjing Xu, Xuliang Li, Hong Zhang, Lei Jin, Shengyong XuSensors 2025 · DOI

TL;DR

This paper introduces the "4D+ City Sidewalk" system, which integrates first-person pedestrian views with overhead CCTV footage to create a detailed, spatiotemporal visualization of urban sidewalks. This system aims to improve pedestrian navigation and urban spatial perception by accurately localizing users and their surroundings, ultimately contributing to more pedestrian-friendly city planning.

What Problem Does This Paper Address?

Modern urban transportation systems often prioritize vehicles over pedestrians, leading to marginalization and increased risks for vulnerable groups like those with mobility impairments or visual disabilities. While intelligent transportation systems (ITS) and visual assistance technologies (VST) offer potential solutions, first-person perspectives have limitations like restricted fields of view. This paper addresses the challenge of providing robust, user-centric urban spatial perception by effectively integrating pedestrian-level data with broader urban sensing infrastructure, specifically leveraging existing street landmarks and CCTV cameras.

Methods

The 4D+ City Sidewalk system operates in two main phases:

  • Spatial Prior Phase: This phase focuses on localizing the user's camera within the sidewalk environment. It uses first-person visual data and identifies common urban landmarks (e.g., streetlights, curbs, tree trunks) to estimate the user's geospatial pose. A 2.5D map, derived from OpenStreetMap data and manually annotated landmark locations, serves as a reference. The system employs a cost function to minimize the dissimilarity between observed features and projected landmark features, iteratively optimizing the camera's extrinsic parameters using a Levenberg-Marquardt algorithm.
  • Target Localization Phase: This phase detects and localizes surrounding mobile participants (pedestrians, cyclists) using high-angle CCTV cameras. Object detection (YOLOv8) and tracking (DeepSort) algorithms are used to monitor targets. Perspective transformation relationships are then applied to map detected objects from the image plane to the spatial coordinate system, leveraging homography matrices derived from CCTV views.

The system then integrates these localized user and target data with the sidewalk map to create a 4D+ virtual monitoring interface, offering a spatiotemporal visualization.

Datasets

  • COCO Dataset: Used for training the YOLOv8n model for object detection.
  • Mapillary Vistas Dataset: Used for training the DeepLabv3 model for semantic segmentation to enable landmark detection.
  • OpenStreetMap (OSM) XML data: Utilized for storing and generating the 2.5D map of landmark locations.

The paper also mentions using data collected from a university campus environment for testing and validation.

Key Insights and Ideas

  • Leveraging Urban Infrastructure: The core idea is to use readily available urban landmarks (street furniture, curbs, etc.) as robust anchors for pedestrian localization, overcoming the limitations of GPS in urban canyons.
  • Synergy of First-Person and Overhead Views: Combining the intimate, user-centric perspective with the broader, contextual view from CCTV cameras provides a more comprehensive understanding of the pedestrian environment.
  • Data Contribution from Users: The system not only provides assistance but also allows users to contribute valuable sidewalk data, enriching urban spatial databases and supporting city planning.
  • 4D+ Concept: The "4D+" signifies the integration of spatiotemporal information (4D) with external data sources and technologies (+), creating a richer, more dynamic representation.

Key Results

  • User Localization Accuracy: The landmark-based localization method achieved an average planar positioning error of 0.468 m and an average height error of 0.120 m.
  • Target Localization Accuracy: With CCTV assistance, the localization of other targets (pedestrians, cyclists) maintained an overall error of 0.24 m.
  • CCTV Advantage: The study highlights that the high-angle CCTV perspective offers higher accuracy (overall error of 0.24 m) and a wider effective range (up to 30 m) compared to the first-person view, especially at greater distances.
  • Qualitative Visualization: The system successfully generates a 3D spatial representation of the user's surroundings, allowing backend personnel to visualize the local spatial structure and positional relationships intuitively.

Limitations and Future Work

  • Occlusion and Low-Light Conditions: The paper acknowledges that vision-based localization faces challenges with occlusions in crowded areas and reduced image quality in low-light or adverse weather conditions.
  • Long-Distance Travel and Processing Demands: For extensive travel and complex outdoor activities, map resource and processing requirements increase.
  • Future Work Suggestions:
    • Hybrid Sensing: Integrating infrared and stereo cameras to address limitations in challenging environments.
  • Cloud-Edge Computing: Exploring cloud-edge models to reduce energy consumption and computational load on pedestrian devices.
  • Real-time Map Updates: Incorporating dynamic updates for temporary blockages or altered pedestrian paths.
  • IoT and Vehicular Integration: Connecting with IoT and vehicular networks for predictive risk assessment and reducing pedestrian-vehicle conflicts.
  • Deployment in Larger Urban Areas: Addressing challenges like low recognition rates under extreme conditions for broader deployment.

Summary updated

Abstract

As urban environments become increasingly interconnected, the demand for precise and efficient pedestrian solutions in digitalized smart cities has grown significantly. This study introduces a scalable spatial visualization system designed to enhance interactions between individuals and the street in outdoor sidewalk environments. The system operates in two main phases: the spatial prior phase and the target localization phase. In the spatial prior phase, the system captures the user's perspective using first-person visual data and leverages landmark elements within the sidewalk environment to localize the user's camera. In the target localization phase, the system detects surrounding objects, such as pedestrians or cyclists, using high-angle closed-circuit television (CCTV) cameras. The system was deployed in a real-world sidewalk environment at an intersection on a university campus. By combining user location data with CCTV observations, a 4D+ virtual monitoring system was developed to present a spatiotemporal visualization of the mobile participants within the user's surrounding sidewalk space. Experimental results show that the landmark-based localization method achieves a planar positioning error of 0.468 m and a height error of 0.120 m on average. With the assistance of CCTV cameras, the localization of other targets maintains an overall error of 0.24 m. This system establishes the spatial relationship between pedestrians and the street by integrating detailed sidewalk views, with promising applications for pedestrian navigation and the potential to enhance pedestrian-friendly urban ecosystems.
BibTeX
@inproceedings{zhao2025d,
  author = {Jinjing Zhao and Yunfan Chen and Yancheng Li and Haotian Xu and Jingjing Xu and Xuliang Li and Hong Zhang and Lei Jin and Shengyong Xu},
  title = {{4D+ City Sidewalk: Integrating Pedestrian View into Sidewalk Spaces to Support User-Centric Urban Spatial Perception}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25051375},
}
Figure