Abstract
In this paper we propose a novel approach to localization in very large indoor spaces (i.e., 200+ store shopping malls) that takes a single image and a floor plan of the environment as input. We formulate the localization problem as inference in a Markov random field, which jointly reasons about text detection (localizing shop's names in the image with precise bounding boxes), shop facade segmentation, as well as camera's rotation and translation within the entire shopping mall. The power of our approach is that it does not use any prior information about appearance and instead exploits text detections corresponding to the shop names. This makes our method applicable to a variety of domains and robust to store appearance variation across countries, seasons, and illumination conditions. We demonstrate the performance of our approach in a new dataset we collected of two very large shopping malls, and show the power of holistic reasoning.
Introduction
Navigating massive indoor environments like shopping malls with hundreds of stores is a common challenge. While outdoor navigation is largely solved by GPS, indoor localization remains difficult due to signal attenuation and the dynamic nature of retail spaces. Existing solutions typically rely on expensive sensor infrastructure (WiFi/Bluetooth beacons) or massive image databases for visual matching. However, visual matching is often brittle in retail environments where storefronts change frequently due to seasonal displays and sales.
This research presents an appearance-free method for monocular localization. Instead of relying on visual features that change over time, the system uses shop names as semantic anchors, matching detected text in a single RGB photo to a 2D vector floor plan—much like a human would navigate using a mall directory.
Methodology
The core of the approach is a Markov Random Field (MRF) that jointly optimizes for text detection, façade segmentation, and camera pose. By reasoning about these elements together, the system ensures geometric and semantic consistency.
Holistic Reasoning in an MRF
The localization problem is formulated as inference in an MRF. The model takes a single image and a floor plan as input and reasons about:
- Text Detection: Identifying shop names (e.g., "GAP") within the image.
- Façade Segmentation: Determining the 2D boundaries of storefronts in the image space.
- Camera Pose: Estimating the 3D rotation and translation of the camera relative to the mall's coordinate system.
Technical Components
The system utilizes a Manhattan World Text Detector to exploit the orthogonal nature of mall corridors. It identifies vanishing points to find candidate text boxes on walls, rectifies them for better OCR performance, and uses a CNN with N-grams for robust recognition. To handle ambiguity in repetitive mall environments, the system generates a ranked list of the "Diverse K-Best" locations rather than a single guess.
The Shopping Mall Dataset
To evaluate the approach, the authors introduced a new benchmark dataset captured in two large Toronto malls (Yorkdale and Promenade). The dataset includes:
- 358 retailers and approximately 16,000 images.
- Ground-truth labels for shop names, 2D façades, and precise 3D camera locations.
- Temporal variety, with data collected months apart to test robustness against seasonal appearance changes.
Results and Insights
The holistic model achieved a 46.43% Recall@1 within a 5-meter threshold, meaning the top guess was highly accurate nearly half the time. When considering the top 5 candidates, the accuracy rose to over 60%.
Ablation studies confirmed that shop name recognition is the most critical feature. Without semantic text information, localization performance dropped significantly, demonstrating that geometric cues alone are insufficient for distinguishing between similar-looking corridors in a large-scale environment.
Conclusion and Future Work
This work demonstrates that text is a powerful, stable anchor for indoor localization. While the system is robust to appearance changes, it currently depends on the visibility of store names and the accuracy of the 3D floor plan. Future work could incorporate additional semantic cues like brand logos or mall directories and extend the framework to handle video sequences for continuous tracking.
Summary updated