TL;DR
This paper introduces LiveMaps, a system that automatically converts static map images found in web search results into interactive map viewports. By analyzing map images to identify the geographical area of interest, LiveMaps allows users to directly explore these locations on online map engines, bridging the gap between static image discovery and interactive exploration.
What Problem Does This Paper Address?
When users search for locations on the web, image search engines often return map images as top results. While these images provide a visual representation of a location, they are static and offer no interactivity. To explore the depicted area further (e.g., zoom in, pan), users are forced to manually issue the same query on a dedicated map search engine. However, this often fails to pinpoint the exact area of interest shown in the original map image, especially if the initial query lacked sufficient context. LiveMaps aims to solve this by automatically annotating map images with the correct interactive map viewport, enabling seamless exploration.
Methods
LiveMaps employs a multi-stage pipeline:
- IsMap Classifier: A deep neural network (ResNet-18) trained to distinguish between map images and general images. This prevents the system from attempting to process non-map images.
- Featurizer: This component extracts relevant information from the map image.
- OCR (Optical Character Recognition): Extracts text directly from the map image. This text is a strong indicator of the geographical area.
- Country Extractor: A visual classifier trained to identify the presence of specific countries based on their distinctive shapes. This is particularly useful when OCR fails or the text is insufficient. These extracted features are combined into a "meta query."
- Viewport Search Engine: This engine uses the "meta query" to search an index of administrative boundaries (countries, cities, states) worldwide. Each entry in the index includes textual information and the geographical shape of the entity. The engine retrieves documents that best match the query and provides a bounding box for the identified location.
- Refinement: To ensure high precision, an Image-Map Similarity classifier is used. It visually compares the original map image with the generated map viewport. If the similarity score is high, the system proceeds; otherwise, it exits without returning a result. This step significantly improves the quality of the final output.
Datasets
- IsMap Classifier Training: Approximately 1,500 annotated map images from ImageNet, plus 50,000 randomly sampled non-map images.
- Country Extractor Training: Automatically labeled images for 250 countries, scraped from queries like "{map of country;}".
- Viewport Search Engine Training: 2,000 locations with their viewports obtained from map search engines.
- Evaluation Dataset (DM): A subset of 3,500 map images identified from a larger set of 20,000 images collected from image search traffic. Judges manually identified these as maps and provided the ground-truth viewport for each.
Key Insights and Ideas
- Bridging the Static-Interactive Gap: The core insight is that users who click on map images in search results likely want to interact with that map. LiveMaps directly addresses this by making the static image actionable.
- Leveraging Multiple Cues: The system effectively combines textual information (OCR) with visual cues (country shapes) to infer the geographical intent, making it robust even when one source of information is weak.
- Importance of Refinement: The Image-Map Similarity classifier is crucial for ensuring that the generated viewport is not just geographically relevant but also visually consistent with the original map image, significantly boosting precision.
- Beyond Image Search: The paper highlights that this technology has broader applications, such as annotating map images embedded in news articles or other web content.
Key Results
The paper demonstrates that LiveMaps significantly outperforms a simple baseline (using the original image search query directly on a map search engine).
- Precision: LiveMaps achieves high precision, especially when the Image-Map Similarity threshold is set high. For example, with a similarity threshold of ≥ 0.9, precision reaches 0.93.
- Recall: While precision is high, recall is lower (0.05 at the highest similarity threshold). This is a trade-off made to ensure high-quality results.
- Qualitative Example: Figure 4 shows a qualitative example where LiveMaps' viewport is a good representation of the Florida Keys, though slightly less zoomed than the human judge's viewport, still achieving an acceptable overlap score (S₁ = 0.307).
Limitations and Future Work
- Recall vs. Precision Trade-off: The current system prioritizes precision, leading to lower recall. Future work aims to improve recall by exploring more feature extractors.
- Country-Specific Focus: The Country Extractor is currently limited to countries. The authors plan to augment it with other locations like islands and states that have distinctive shapes.
- Cropping Improvement: The paper suggests that LiveMaps could be improved by cropping the returned administrative bounding box to more accurately match the original image's viewport.
- Expanding Feature Extractors: The system currently uses OCR and Country Detection. Future work will focus on incorporating more feature extractors to enhance the system's ability to understand map content.
Summary updated