A novel fusion method for integrating multiple modalities and knowledge for multimodal location estimation

Pascal Kelm, Sebastian Schmiedeke, Jaeyoung Choi, Gerald Friedland, Venkatesan Nallampatti Ekambaram, Kannan Ramchandran, Thomas SikoraGeoMM@ACM Multimedia 2013 · DOI

TL;DR

This paper introduces a novel fusion method that significantly improves the accuracy of automatically estimating the geographical location of multimedia content (like videos) that lacks GPS tags. By integrating visual and textual information with external knowledge bases, their approach achieves state-of-the-art performance, locating a substantial portion of videos within a 100m radius.

What Problem Does This Paper Address?

In today's digital world, multimedia content is abundant, but a significant portion lacks crucial metadata like GPS coordinates. This absence hinders effective organization, retrieval, and understanding of this content. While some devices automatically tag media, the vast majority of user-generated content remains untagged. This paper tackles the challenge of "placing" or "geo-tagging" this untagged multimedia, specifically focusing on videos, by developing a method to automatically infer their likely geographical origin. This is particularly important for content captured indoors or in areas with poor GPS reception, and for organizing personal media archives like vacation photos and videos.

Methods

The core of the paper's contribution lies in a novel fusion framework that integrates multiple data sources and knowledge bases in a hierarchical manner. Here's a breakdown of their key techniques:

  • Hierarchical Spatial Segmentation: The world map is iteratively divided into increasingly finer spatial segments. This hierarchical approach allows for efficient processing by focusing on smaller regions at lower levels. These segments are generated using two main methods:
  • National Borders Detection: This method leverages textual metadata (titles, descriptions, keywords) to identify toponyms (place names). These toponyms are then used to query gazetteers (like GeoNames) and Wikipedia to confirm their geographical relevance and narrow down potential locations to specific countries. This step is crucial for disambiguating place names and reducing the search space.
  • Spatial Grids: In addition to national borders, a grid-based segmentation of the map is used, with grids of varying sizes to achieve different levels of granularity.
  • Multimodal Feature Integration: The framework processes textual and visual modalities separately before fusing them:
  • Textual Approach: A probabilistic multinomial Bag-of-Words model is employed. Textual metadata is processed, stemmed, and then used to calculate term-location probabilities. Different weighting schemes (Term Occurrence, Term Frequency, and Term Frequency-Inverse Document Frequency - TF-IDF) are explored, with TF-IDF showing the best performance.
  • Visual Approach: Visual features extracted from video keyframes are used to predict location. A k-d tree is built for efficient nearest neighbor search to find visually similar spatial segments.
  • Centroid-based Candidate Fusion: To address data sparsity and combine the strengths of different modalities, a centroid-based fusion method is proposed. This approach calculates a weighted average of candidate locations from both textual and visual modalities. The weights are derived from the confidence scores of each modality. This fusion method allows for a more robust location estimation, even when individual modalities are weak or data is sparse.

Datasets

The primary dataset used for evaluation is the MediaEval 2010 Placing Task dataset. This dataset consists of consumer-produced videos that were either geo-tagged or used for evaluating geo-tagging algorithms. The authors also mention using a geo-tagged database for training their models.

Key Insights and Ideas

  • Textual information is powerful, but needs context: While textual metadata alone can provide good location estimates, generic terms (like "911") can be surprisingly informative when combined with other context, even if they aren't direct geographical entries.
  • Visual features alone are weak: The paper highlights that purely visual approaches show low correlation with locations and achieve lower precision than tag-based methods. However, when combined with a toponym lookup (like national borders detection), even weak visual information can improve geo-tagging performance.
  • Hierarchical approach is key for efficiency and accuracy: Breaking down the problem into hierarchical spatial segments significantly reduces computational cost and allows for progressively finer location estimations.
  • Fusion is crucial for robustness: Combining textual and visual modalities, especially through a centroid-based fusion method, is essential for overcoming the limitations of individual modalities and handling data sparsity.
  • National borders detection is a strong prior: Using national borders derived from textual metadata acts as a powerful pre-selection mechanism, significantly improving accuracy by reducing ambiguity.

Key Results

  • The proposed fusion method significantly outperforms other state-of-the-art approaches on the MediaEval 2010 Placing Task dataset.
  • The system successfully locates approximately 40% of the videos to within a radius of 100 meters.
  • For a margin of error of 1 km, the system achieves an accuracy of 78.5%.
  • The TF-IDF weighting scheme for the textual approach yields better results than Term Occurrence or Term Frequency alone.
  • The national borders detection significantly boosts accuracy, achieving 86.8% accuracy within a 10 km radius when used with TF-IDF weighting.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their approach significantly increases the number of geo-tagged media, there's still room for improvement. The performance is also dependent on the quality and presence of textual metadata.
  • Future Work: The authors explicitly state their plan to work on adding acoustic location estimation as another modality to their fusion framework.

Summary updated

BibTeX
@inproceedings{kelm2013novel,
  author = {Pascal Kelm and Sebastian Schmiedeke and Jaeyoung Choi and Gerald Friedland and Venkatesan Nallampatti Ekambaram and Kannan Ramchandran and Thomas Sikora},
  title = {{A novel fusion method for integrating multiple modalities and knowledge for multimodal location estimation}},
  booktitle = {GeoMM@ACM Multimedia},
  year = {2013},
  doi = {10.1145/2509230.2509238},
}
Figure