TL;DR
StreetNav is a novel navigation system that repurposes existing urban street cameras to provide high-precision, real-time guidance for blind and low-vision (BLV) pedestrians. By combining a server-side computer vision pipeline with a smartphone app, the system achieves centimeter-level localization and provides scene-aware assistance—such as obstacle alerts and street-crossing timings—that far outperforms traditional GPS-based solutions.
What Problem Does This Paper Address?
Outdoor navigation is a significant challenge for the BLV community. While GPS-based apps are common, they suffer from "urban canyon" effects where tall buildings and trees degrade signal accuracy to 10–15 meters. This imprecision leads to "veering" (wandering off-path), difficulty finding exact building entrances, and safety risks during street crossings.
Existing high-precision solutions often require users to purchase expensive wearable hardware or perform cumbersome actions, like pointing their phone camera at specific landmarks. StreetNav explores a different path: Infrastructure-as-a-Service. It asks whether the thousands of cameras already monitoring our streets can be leveraged to support accessibility without placing the sensing or financial burden on the user.
Methods
StreetNav consists of two primary components: a server-side Computer Vision (CV) pipeline and a companion smartphone app.
- Localization & Tracking: The system uses YOLOv8 for object detection and Nvidia’s DCF-based multi-object tracker to follow pedestrians in the camera feed. To distinguish the BLV user from the crowd, the authors introduced a "gesture-based login": the user waves a hand above their head, which is recognized by a CLIP-based model.
- Mapping & Path Planning: The system performs a homography transformation to project the user's coordinates from the 2D camera view onto a top-down satellite map. Navigation is handled by an A* algorithm running on a manually annotated graph of the intersection (see Figure 4).
- Scene Awareness: The pipeline identifies obstacles (cars, poles, other pedestrians) within a 4-foot radius of the user. It also monitors pedestrian signals by analyzing pixel color thresholds (walk vs. wait) in specific regions of interest.
- Feedback Loop: The app provides audio-haptic feedback. Spatialized beeps indicate veering (e.g., beeping in the right ear if the user veers left), while haptic vibrations provide positive reinforcement when the user is on track.
Datasets
The researchers utilized the NSF PAWR COSMOS testbed, a real-world smart city infrastructure located in Harlem, NYC.
- Technical Evaluation: Based on a 15-minute video (22,500 frames) recorded from a second-story street camera.
- Manual Annotations: The team annotated 140 image crops for gesture recognition, 250 frames for pedestrian feet position ground truth, and 50 points for camera-to-map transformation calibration.
Key Insights and Ideas
- Sociotechnical Privacy: A key "aha moment" came from stakeholder interviews. Residents and BLV users were significantly more comfortable with repurposing existing cameras than installing new ones. They viewed the dual-use (safety + accessibility) as a fair trade-off, provided data was processed in real-time and not stored indefinitely.
- Infrastructure Sensing: By moving the "eyes" from the user's hand to a fixed overhead position, the system eliminates the need for users to point their phones at signals or obstacles, allowing them to keep their hands free for white canes or guide dogs.
- Socially Acceptable Interaction: The hand-waving gesture was found to be a socially acceptable and effective way to establish a connection with public infrastructure, though some participants noted it might be misinterpreted by passersby as a call for help.
Key Results
The system was evaluated with 8 BLV participants against BlindSquare, a popular GPS-based app.
- Precision: StreetNav guided users to within a radius 2.9x tighter to their destination than the GPS baseline.
- Efficiency: Users traveled 51% less unnecessary distance compared to using GPS, avoiding the common "oscillatory" patterns seen when GPS signals drift near a destination.
- Veering: The audio-haptic feedback reduced veering off-course by 53%.
- Technical Accuracy: The system achieved a localization error of approximately 1.06 meters (0.41m for feet estimation + 0.65m for map transformation), a massive improvement over the 10-15m errors typical of urban GPS.
- User Preference: 100% of study participants preferred StreetNav over the baseline.
Limitations and Future Work
- Scalability: The current study was limited to a single intersection. Real-world deployment would require a network of cameras with seamless "hand-offs" as a user moves between fields of view.
- Environmental Sensitivity: The CV pipeline's performance degrades in low-light conditions or when large objects (like buses or construction scaffolding) occlude the user. The false negative rate for detection jumped from 1% at 5 meters to 74% at 40 meters.
- Privacy & Governance: While real-time processing mitigates some concerns, the question of who owns the data and how to prevent government surveillance overreach remains a complex hurdle for city-wide adoption.
Summary updated