StreetNav: Leveraging Street Cameras to Support Precise Outdoor Navigation for Blind Pedestrians

Gaurav Jain, Basel Hindi, Zihao Zhang, Koushik Srinivasula, Mingyu Xie, Mahshid Ghasemi, Daniel Weiner, Sophie Ana Paris, Xin Yi Therese Xu, Michael Malcolm, Mehmet Turkcan, Javad Ghaderi, Zoran Kostic, Gil Zussman, Brian A. Smith

TL;DR

StreetNav is a novel navigation system that repurposes existing urban street cameras to provide high-precision, real-time guidance for blind and low-vision (BLV) pedestrians. By combining a server-side computer vision pipeline with a smartphone app, the system achieves centimeter-level localization and provides scene-aware assistance—such as obstacle alerts and street-crossing timings—that far outperforms traditional GPS-based solutions.

What Problem Does This Paper Address?

Outdoor navigation is a significant challenge for the BLV community. While GPS-based apps are common, they suffer from "urban canyon" effects where tall buildings and trees degrade signal accuracy to 10–15 meters. This imprecision leads to "veering" (wandering off-path), difficulty finding exact building entrances, and safety risks during street crossings.

Existing high-precision solutions often require users to purchase expensive wearable hardware or perform cumbersome actions, like pointing their phone camera at specific landmarks. StreetNav explores a different path: Infrastructure-as-a-Service. It asks whether the thousands of cameras already monitoring our streets can be leveraged to support accessibility without placing the sensing or financial burden on the user.

Methods

StreetNav consists of two primary components: a server-side Computer Vision (CV) pipeline and a companion smartphone app.

  1. Localization & Tracking: The system uses YOLOv8 for object detection and Nvidia’s DCF-based multi-object tracker to follow pedestrians in the camera feed. To distinguish the BLV user from the crowd, the authors introduced a "gesture-based login": the user waves a hand above their head, which is recognized by a CLIP-based model.
  2. Mapping & Path Planning: The system performs a homography transformation to project the user's coordinates from the 2D camera view onto a top-down satellite map. Navigation is handled by an A* algorithm running on a manually annotated graph of the intersection (see Figure 4).
  3. Scene Awareness: The pipeline identifies obstacles (cars, poles, other pedestrians) within a 4-foot radius of the user. It also monitors pedestrian signals by analyzing pixel color thresholds (walk vs. wait) in specific regions of interest.
  4. Feedback Loop: The app provides audio-haptic feedback. Spatialized beeps indicate veering (e.g., beeping in the right ear if the user veers left), while haptic vibrations provide positive reinforcement when the user is on track.
Figure 1: StreetNav is a system that explores the concept of repurposing existing street cameras to support precise outdoor navigation for blind and low-vision (BLV) pedestrians. It comprises two components: (i) a computer vision (CV) pipeline, and (ii) a companion smartphone app. The computer vision pipeline processes the street camera’s video feeds and delivers real-time navigation feedback via the app. StreetNav offers precise turn-by-turn directions to destinations while also providing real-time, scene-aware assistance to alert them of nearby obstacles and facilitate safe street crossings.

Datasets

The researchers utilized the NSF PAWR COSMOS testbed, a real-world smart city infrastructure located in Harlem, NYC.

  • Technical Evaluation: Based on a 15-minute video (22,500 frames) recorded from a second-story street camera.
  • Manual Annotations: The team annotated 140 image crops for gesture recognition, 250 frames for pedestrian feet position ground truth, and 50 points for camera-to-map transformation calibration.

Key Insights and Ideas

  • Sociotechnical Privacy: A key "aha moment" came from stakeholder interviews. Residents and BLV users were significantly more comfortable with repurposing existing cameras than installing new ones. They viewed the dual-use (safety + accessibility) as a fair trade-off, provided data was processed in real-time and not stored indefinitely.
  • Infrastructure Sensing: By moving the "eyes" from the user's hand to a fixed overhead position, the system eliminates the need for users to point their phones at signals or obstacles, allowing them to keep their hands free for white canes or guide dogs.
  • Socially Acceptable Interaction: The hand-waving gesture was found to be a socially acceptable and effective way to establish a connection with public infrastructure, though some participants noted it might be misinterpreted by passersby as a call for help.
Figure 4: StreetNav’s internal graph representation for route planning. The user’s current position is added dynamically as a start node to the graph upon choosing a destination. The shortest path, highlighted in green, is then calculated as per this graph representation.

Key Results

The system was evaluated with 8 BLV participants against BlindSquare, a popular GPS-based app.

  • Precision: StreetNav guided users to within a radius 2.9x tighter to their destination than the GPS baseline.
  • Efficiency: Users traveled 51% less unnecessary distance compared to using GPS, avoiding the common "oscillatory" patterns seen when GPS signals drift near a destination.
  • Veering: The audio-haptic feedback reduced veering off-course by 53%.
  • Technical Accuracy: The system achieved a localization error of approximately 1.06 meters (0.41m for feet estimation + 0.65m for map transformation), a massive improvement over the 10-15m errors typical of urban GPS.
  • User Preference: 100% of study participants preferred StreetNav over the baseline.

Limitations and Future Work

  • Scalability: The current study was limited to a single intersection. Real-world deployment would require a network of cameras with seamless "hand-offs" as a user moves between fields of view.
  • Environmental Sensitivity: The CV pipeline's performance degrades in low-light conditions or when large objects (like buses or construction scaffolding) occlude the user. The false negative rate for detection jumped from 1% at 5 meters to 74% at 40 meters.
  • Privacy & Governance: While real-time processing mitigates some concerns, the question of who owns the data and how to prevent government surveillance overreach remains a complex hurdle for city-wide adoption.

Summary updated

Abstract

Blind and low-vision (BLV) people rely on GPS-based systems for outdoor navigation. GPS's inaccuracy, however, causes them to veer off track, run into obstacles, and struggle to reach precise destinations. While prior work has made precise navigation possible indoors via hardware installations, enabling this outdoors remains a challenge. Interestingly, many outdoor environments are already instrumented with hardware such as street cameras. In this work, we explore the idea of repurposing existing street cameras for outdoor navigation. Our community-driven approach considers both technical and sociotechnical concerns through engagements with various stakeholders: BLV users, residents, business owners, and Community Board leadership. The resulting system, StreetNav, processes a camera's video feed using computer vision and gives BLV pedestrians real-time navigation assistance. Our evaluations show that StreetNav guides users more precisely than GPS, but its technical performance is sensitive to environmental occlusions and distance from the camera. We discuss future implications for deploying such systems at scale.
BibTeX
@inproceedings{jain2023streetnav,
  author = {Gaurav Jain and Basel Hindi and Zihao Zhang and Koushik Srinivasula and Mingyu Xie and Mahshid Ghasemi and Daniel Weiner and Sophie Ana Paris and Xin Yi Therese Xu and Michael Malcolm and Mehmet Turkcan and Javad Ghaderi and Zoran Kostic and Gil Zussman and Brian A. Smith},
  title = {{StreetNav: Leveraging Street Cameras to Support Precise Outdoor Navigation for Blind Pedestrians}},
  booktitle = {UIST},
  year = {2023},
}
Figure