An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval

Lei Zhu, Jun Long, Chengyuan Zhang, Ruipeng Chen, Xinpan Yuan, Zhan YangIEEE Access 2018 · OpenReview

TL;DR

This paper introduces "geo-multimedia cross-modal retrieval," a new way to search for multimedia content (like images and text) that considers both its geographical location and semantic meaning. They propose a novel framework, including a cross-modal semantic matching method and a hybrid indexing structure (GMR-Tree), to efficiently retrieve relevant results by bridging the semantic gap between different modalities.

What Problem Does This Paper Address?

The explosion of location-aware mobile devices and online services has led to a massive influx of multimedia data tagged with geographical information (geo-multimedia). Existing retrieval methods struggle to effectively handle this data because they either focus on traditional cross-modal retrieval (ignoring geo-tags) or spatial keyword search (ignoring multimedia content). This paper addresses the challenge of finding geo-multimedia objects that are both geographically close to a query location and semantically similar to a query in a different modality (e.g., finding images based on a text description near a specific location).

Methods

The authors propose a comprehensive framework for geo-multimedia cross-modal retrieval. Key components include:

  • kNN Geo-Multimedia Cross-Modal Query: A new query definition that explicitly considers both geographical proximity and semantic similarity.
  • Cross-Modal Semantic Representation Space (CoSReS): A novel concept to bridge the semantic gap between different modalities by mapping them into a common, high-level semantic abstraction.
  • Cross-Modal Semantic Matching (CoSMat): A method to construct CoSReS, comprising two parts:
  • CorrProj: Uses Canonical Correlation Analysis (CCA) to learn maximally correlated subspaces for text and image features.
  • LogsTran: Employs logistic regression to learn semantic mappings within these subspaces, creating isomorphic representation spaces.
  • Deep Learning for Feature Extraction: Utilizes Convolutional Neural Networks (CNNs) for visual features and Latent Dirichlet Allocation (LDA) for textual features to capture high-level semantic information.
  • Geo-Multimedia R-Tree (GMR-Tree): A novel hybrid indexing structure that combines R-Trees for spatial indexing with signatures representing semantic information of geo-multimedia objects.
  • kGMCMS Algorithm: An efficient kNN search algorithm designed to leverage the GMR-Tree for fast geo-multimedia cross-modal retrieval.

Datasets

  • Real Dataset: Over one million geo-tagged images crawled from Flickr.
  • Synthetic Dataset: Generated by obtaining spatial locations from Rtree-Portal and randomly geo-tagging these locations with images from ImageNet. ImageNet is described as a large-scale hierarchical image database organized by WordNet.

Key Insights and Ideas

  • The "Geo-Multimedia" Gap: The paper highlights that simply combining existing cross-modal retrieval and spatial search techniques isn't sufficient. The unique challenge lies in simultaneously considering both spatial and semantic relevance for multimedia content.
  • Bridging the Semantic Gap: The concept of a "Cross-Modal Semantic Representation Space" (CoSReS) is a crucial idea. By projecting different modalities into a common semantic space, semantic similarity can be directly measured, overcoming the inherent differences in feature representations.
  • Hybrid Indexing is Key for Efficiency: The GMR-Tree, by integrating spatial (MBR) and semantic (signature) information, is designed to significantly speed up the search process compared to traditional spatial indexing methods.

Key Results

  • Efficiency: Experiments on both real and synthetic datasets show that the proposed DeCoSReS+GMR-Tree method achieves the smallest response times compared to baseline methods (DeCoSReS+R-Tree and SM+R-Tree). The performance advantage of GMR-Tree becomes more pronounced with larger datasets and a higher number of requested results.
  • Precision: On the real dataset, the DeCoSReS+GMR-Tree method shows slightly higher precision than the SM+R-Tree baseline, attributed to the more discriminative features extracted by CNNs compared to SIFT+BoVW.

Limitations and Future Work

  • Limitations: The paper focuses on two modalities (text and image). While the framework is presented as generalizable, the current implementation and evaluation are limited to these two. The paper also acknowledges that the semantic representation space construction is a complex problem.
  • Future Work: The authors suggest that their framework can be extended to more modalities. The core problem of efficiently learning intermediate representation spaces for cross-modal retrieval remains a significant area for future research.

Summary updated

Abstract

Due to the rapid development of mobile Internet techniques, cloud computation and popularity of online social networking and location-based services, massive amount of multimedia data with geographical information is generated and uploaded to the Internet. In this paper, we propose a novel type of cross-modal multimedia retrieval called geo-multimedia cross-modal retrieval which aims to search out a set of geo-multimedia objects based on geographical distance proximity and semantic similarity between different modalities. Previous studies for cross-modal retrieval and spatial keyword search cannot address this problem effectively because they do not consider multimedia data with geo-tags and do not focus on this type of query. In order to address this problem efficiently, we present the definition of $k$NN geo-multimedia cross-modal query at the first time and introduce relevant conceptions such as cross-modal semantic representation space. To bridge the semantic gap between different modalities, we propose a method named cross-modal semantic matching which contains two important component, i.e., CorrProj and LogsTran, which aims to construct a common semantic representation space for cross-modal semantic similarity measurement. Besides, we designed a framework based on deep learning techniques to implement common semantic representation space construction. In addition, a novel hybrid indexing structure named GMR-Tree combining geo-multimedia data and R-Tree is presented and a efficient $k$NN search algorithm called $k$GMCMS is designed. Comprehensive experimental evaluation on real and synthetic dataset clearly demonstrates that our solution outperforms the-state-of-the-art methods.
BibTeX
@inproceedings{zhu2018efficient,
  author = {Lei Zhu and Jun Long and Chengyuan Zhang and Ruipeng Chen and Xinpan Yuan and Zhan Yang},
  title = {{An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval}},
  booktitle = {IEEE Access},
  year = {2018},
  url = {https://openreview.net/forum?id=Eht2cjmGGM},
}
Figure