Introduction
Accurately mapping the world’s businesses is a monumental task. While GPS provides the coordinates of a building, it fails to define the precise boundaries of a storefront or identify the specific nature of the business. Researchers from Google Street View have developed an end-to-end deep learning system designed to automatically detect business storefronts from millions of panoramic street-level images. By transitioning from traditional object proposal methods to a "MultiBox" approach, they achieved human-level precision and a 37x speedup, enabling automated mapping at a global scale.
The Challenge of Storefront Detection
Detecting storefronts is significantly more complex than standard object detection for several reasons:
- Extreme Variability: Storefronts vary wildly across different regions and industries. A gas station in the suburbs looks nothing like a boutique in Tokyo or a dry cleaner in New York.
- Ambiguous Boundaries: In dense urban environments, businesses are often packed side-by-side without clear physical gaps, making it difficult to define where one bounding box should end and the next begin.
- Environmental Noise: Street-level imagery is frequently affected by motion blur, reflections on glass, occlusions from trees or vehicles, and inconsistent lighting.
Traditional computer vision methods, such as Selective Search, often struggle in these scenarios because they rely on low-level cues like color or texture gradients. These cues are often insufficient to distinguish between two adjacent man-made structures.
Methodology: The MultiBox Approach
The core of the proposed system is MultiBox, a technique that replaces the traditional "proposal then classify" workflow with a single Convolutional Neural Network (CNN).
1. Learned Proposals
Instead of searching for "interesting" regions using hand-crafted algorithms, MultiBox utilizes a set of 800 "priors"—precomputed bounding box locations based on common storefront shapes. The network takes raw pixels as input and directly predicts which priors best match the objects in the image, providing a confidence score for each.
2. Multi-Crop Evaluation and Post-Classification
Because Street View panoramas are high-resolution, the system employs a coarse sliding window (87 crops per panorama) to ensure small businesses are not overlooked. To further refine accuracy, the top MultiBox detections are passed through a GoogLeNet classifier. This secondary step improves average precision by nearly 7% by confirming whether a detected object is indeed a business.
3. Handling Distortion
Rather than computationally "flattening" the spherical distortion inherent in panoramas, the researchers trained the network to recognize objects directly within the equirectangular projection, increasing efficiency.
Data and Scale
The scale of the training and evaluation data is a key highlight of this research:
- Training Set: Approximately 2 million panorama images from 12 different countries, labeled via crowdsourcing.
- Evaluation Set: A "gold standard" set of 2,000 panoramas containing roughly 11,000 annotations.
- Real-World Benchmark: A 1-square-kilometer area in Brazil was exhaustively mapped by hand to create a ground-truth baseline for "business discovery" in a live environment.
Key Insights and Results
The transition to learned proposals proved to be a major breakthrough. While Selective Search achieved only 62% recall with over 4,600 proposals, MultiBox reached 91% recall with only 863 proposals. Furthermore, the system brought processing time down from 1.5 minutes per image to just 2.5 seconds per panorama, a 37x speedup that makes worldwide deployment feasible.
In the Brazil benchmark, the system achieved 94.6% precision, successfully identifying 495 unique businesses. Interestingly, the researchers found that human annotators disagree on business boundaries more than 10% of the time, suggesting that the "Auto Detector" has reached or even exceeded human-level precision in certain settings.
Future Directions
While highly effective, the system currently has a recall rate of approximately 53.2% in real-world tests, meaning it misses about half of the existing businesses. Future work may involve incorporating more global context—such as recognizing that a storefront is unlikely to be located on a high-speed highway—to filter out false positives like advertisements on trucks. Additionally, moving to deeper neural networks could further improve the delineation of complex boundaries.
Summary updated