TL;DR
This paper introduces the "4D+ City Sidewalk" system, which integrates first-person pedestrian views with overhead CCTV footage to create a detailed, spatiotemporal visualization of urban sidewalks. This system aims to improve pedestrian navigation and urban spatial perception by accurately localizing users and their surroundings, ultimately contributing to more pedestrian-friendly city planning.
What Problem Does This Paper Address?
Modern urban transportation systems often prioritize vehicles over pedestrians, leading to marginalization and increased risks for vulnerable groups like those with mobility impairments or visual disabilities. While intelligent transportation systems (ITS) and visual assistance technologies (VST) offer potential solutions, first-person perspectives have limitations like restricted fields of view. This paper addresses the challenge of providing robust, user-centric urban spatial perception by effectively integrating pedestrian-level data with broader urban sensing infrastructure, specifically leveraging existing street landmarks and CCTV cameras.
Methods
The 4D+ City Sidewalk system operates in two main phases:
- Spatial Prior Phase: This phase focuses on localizing the user's camera within the sidewalk environment. It uses first-person visual data and identifies common urban landmarks (e.g., streetlights, curbs, tree trunks) to estimate the user's geospatial pose. A 2.5D map, derived from OpenStreetMap data and manually annotated landmark locations, serves as a reference. The system employs a cost function to minimize the dissimilarity between observed features and projected landmark features, iteratively optimizing the camera's extrinsic parameters using a Levenberg-Marquardt algorithm.
- Target Localization Phase: This phase detects and localizes surrounding mobile participants (pedestrians, cyclists) using high-angle CCTV cameras. Object detection (YOLOv8) and tracking (DeepSort) algorithms are used to monitor targets. Perspective transformation relationships are then applied to map detected objects from the image plane to the spatial coordinate system, leveraging homography matrices derived from CCTV views.
The system then integrates these localized user and target data with the sidewalk map to create a 4D+ virtual monitoring interface, offering a spatiotemporal visualization.
Datasets
- COCO Dataset: Used for training the YOLOv8n model for object detection.
- Mapillary Vistas Dataset: Used for training the DeepLabv3 model for semantic segmentation to enable landmark detection.
- OpenStreetMap (OSM) XML data: Utilized for storing and generating the 2.5D map of landmark locations.
The paper also mentions using data collected from a university campus environment for testing and validation.
Key Insights and Ideas
- Leveraging Urban Infrastructure: The core idea is to use readily available urban landmarks (street furniture, curbs, etc.) as robust anchors for pedestrian localization, overcoming the limitations of GPS in urban canyons.
- Synergy of First-Person and Overhead Views: Combining the intimate, user-centric perspective with the broader, contextual view from CCTV cameras provides a more comprehensive understanding of the pedestrian environment.
- Data Contribution from Users: The system not only provides assistance but also allows users to contribute valuable sidewalk data, enriching urban spatial databases and supporting city planning.
- 4D+ Concept: The "4D+" signifies the integration of spatiotemporal information (4D) with external data sources and technologies (+), creating a richer, more dynamic representation.
Key Results
- User Localization Accuracy: The landmark-based localization method achieved an average planar positioning error of 0.468 m and an average height error of 0.120 m.
- Target Localization Accuracy: With CCTV assistance, the localization of other targets (pedestrians, cyclists) maintained an overall error of 0.24 m.
- CCTV Advantage: The study highlights that the high-angle CCTV perspective offers higher accuracy (overall error of 0.24 m) and a wider effective range (up to 30 m) compared to the first-person view, especially at greater distances.
- Qualitative Visualization: The system successfully generates a 3D spatial representation of the user's surroundings, allowing backend personnel to visualize the local spatial structure and positional relationships intuitively.
Limitations and Future Work
- Occlusion and Low-Light Conditions: The paper acknowledges that vision-based localization faces challenges with occlusions in crowded areas and reduced image quality in low-light or adverse weather conditions.
- Long-Distance Travel and Processing Demands: For extensive travel and complex outdoor activities, map resource and processing requirements increase.
- Future Work Suggestions:
- Hybrid Sensing: Integrating infrared and stereo cameras to address limitations in challenging environments.
- Cloud-Edge Computing: Exploring cloud-edge models to reduce energy consumption and computational load on pedestrian devices.
- Real-time Map Updates: Incorporating dynamic updates for temporary blockages or altered pedestrian paths.
- IoT and Vehicular Integration: Connecting with IoT and vehicular networks for predictive risk assessment and reducing pedestrian-vehicle conflicts.
- Deployment in Larger Urban Areas: Addressing challenges like low recognition rates under extreme conditions for broader deployment.
Summary updated