Visual Localizer: Outdoor Localization Based on ConvNet Descriptor and Global Optimization for Visually Impaired Pedestrians

Shufei Lin, Ruiqi Cheng, Kaiwei Wang, Kailun Yang

TL;DR

This paper introduces "Visual Localizer," a novel system for robust outdoor visual localization specifically designed for visually impaired pedestrians. By combining a carefully selected ConvNet descriptor with a global optimization approach using network flow, it achieves accurate localization despite significant environmental changes like viewpoint, illumination, and seasonal variations, outperforming traditional GNSS methods.

What Problem Does This Paper Address?

Visually impaired individuals face significant challenges with outdoor navigation. While GNSS (Global Navigation Satellite System) offers a coarse localization, its typical 10-20 meter error is insufficient for safe and independent navigation, especially in complex environments like intersections. Existing visual localization methods, often developed for autonomous vehicles, are not directly applicable to assistive navigation due to the unique challenges posed by wearable cameras: significant viewpoint variations, dynamic object occlusions, and the need for highly precise localization. This paper aims to bridge this gap by developing a robust visual localization system tailored for the specific needs of visually impaired pedestrians.

Methods

The proposed "Visual Localizer" system has two main components:

  1. ConvNet-Based Image Representation: The authors conducted a comprehensive comparison of five popular Convolutional Neural Networks (AlexNet, VGG16, GoogLeNet, SqueezeNet, and MobileNet) to identify the most suitable descriptor for visual localization. They analyzed the performance of individual layers from these networks across various environmental changes (viewpoint, illumination, cross-season). Ultimately, they found that GoogLeNet, particularly a combination of its Inception3a/3x3 and Inception3a/3x3_reduce layers, offered the best balance of robustness and computational efficiency. To further optimize this descriptor, they employed a compression technique, reducing the descriptor size from hundreds of thousands to a few thousand bytes while retaining high performance.
  2. Global Optimization of Image Matching: To enhance the robustness of matching query images to a database, the system utilizes a data association graph. This graph represents potential matches between query and database images, and a minimum-cost flow problem is solved on this graph to find the globally optimal set of matches. This approach effectively filters out incorrect matches and improves overall localization accuracy, especially in challenging scenarios.

Datasets

The authors evaluated their system on several publicly available datasets and also collected data from visually impaired volunteers:

  • Gardens Point Walking dataset: Features viewpoint changes (left vs. right) and illumination changes (day vs. night).
  • Nordland dataset: Used for testing cross-season changes (summer vs. winter) with no significant viewpoint variations.
  • Bonn dataset: Includes illumination and viewpoint changes, with revisits to the same places.
  • Freiburg dataset: Captures significant perceptual and structural changes over three years, including viewpoint and extreme seasonal variations.
  • Customized dataset: Images captured by visually impaired volunteers in real-world environments (Yuquan Campus of Zhejiang University and West Lake area in Hangzhou, China). This dataset is crucial for validating the system in practical, unconstrained scenarios.

Key Insights and Ideas

  • Mid-level ConvNet features are key: The study revealed that mid-level convolutional layers (e.g., conv3 in AlexNet) often provide the most robust features against appearance changes, contrary to the intuition that deeper layers are always better.
  • Filter size isn't everything: Interestingly, smaller filter sizes (like 1x1) in convolutional layers, especially when combined effectively, can lead to better appearance and viewpoint invariance.
  • Object recognition accuracy doesn't directly translate to place recognition: ConvNets that excel at object classification (like VGG16 and MobileNet) don't necessarily perform best for visual place localization, highlighting the distinct nature of these tasks.
  • GoogLeNet's Inception modules are powerful: The Inception architecture's ability to use filters of different sizes and combine their outputs proved highly effective for robust visual localization, offering the best overall performance.
  • Global optimization is crucial: Simply finding the best single image match is insufficient. The network flow-based global optimization significantly improves robustness by considering the relationships between multiple matches and filtering out false positives.
  • Compression is vital for real-world deployment: Reducing the size of ConvNet descriptors is essential for practical applications, and the authors demonstrated that significant compression can be achieved with minimal performance degradation.

Key Results

  • GoogLeNet selected as optimal: GoogLeNet consistently outperformed other ConvNets in appearance and viewpoint invariance, with modest computational complexity.
  • Descriptor compression: The system achieved a 95.34% reduction in descriptor size (from 175,616 to 8192 dimensions) with only a slight decrease in F1-score (from 0.8406 to 0.8131).
  • Robustness demonstrated: Experiments showed Visual Localizer to be robust against viewpoint changes, illumination changes (dim vs. bright light), cross-season changes (winter vs. summer), and even route changes (correctly failing to match new, unrecorded routes).
  • Superior to GNSS: In real-world experiments, Visual Localizer achieved significantly lower mean error (7.95 vs. 24.09) and higher precision (89.66% vs. 39.22%) compared to GNSS-based localization, despite retrieving fewer matched positions.

Limitations and Future Work

  • Offline processing: Currently, the image matching is performed offline. The authors plan to develop an effective online image matching system.
  • Audio feedback: Designing audio feedback for the Visual Localizer on the Intoer system is a future goal.
  • Autonomous labeling: Future work includes understanding visual information, autonomously labeling key places, and recognizing multiple images of the same place as a single entity.
  • Real-time performance: While the compressed descriptor improves efficiency, achieving real-time performance for online applications will likely require further optimization.

Summary updated

Abstract

Localization systems play an important role in assisted navigation. Precise localization renders visually impaired people aware of ambient environments and prevents them from coming across potential hazards. The majority of visual localization algorithms, which are applied to autonomous vehicles, are not adaptable completely to the scenarios of assisted navigation. Those vehicle-based approaches are vulnerable to viewpoint, appearance and route changes (between database and query images) caused by wearable cameras of assistive devices. Facing these practical challenges, we propose Visual Localizer, which is composed of ConvNet descriptor and global optimization, to achieve robust visual localization for assisted navigation. The performance of five prevailing ConvNets are comprehensively compared, and GoogLeNet is found to feature the best performance on environmental invariance. By concatenating two compressed convolutional layers of GoogLeNet, we use only thousands of bytes to represent image efficiently. To further improve the robustness of image matching, we utilize the network flow model as a global optimization of image matching. The extensive experiments using images captured by visually impaired volunteers illustrate that the system performs well in the context of assisted navigation.
BibTeX
@inproceedings{lin2018visual,
  author = {Shufei Lin and Ruiqi Cheng and Kaiwei Wang and Kailun Yang},
  title = {{Visual Localizer: Outdoor Localization Based on ConvNet Descriptor and Global Optimization for Visually Impaired Pedestrians}},
  booktitle = {Sensors},
  year = {2018},
  url = {https://openreview.net/forum?id=YfyH0GsNV-Ni},
}
Figure