A novel cross-modal hashing algorithm based on multimodal deep learning

Wen Qu, Daling Wang, Shi Feng, Yifei Zhang, Ge Yu

Abstract

With the growing popularity of multimodal data on the Web, cross-modal retrieval on large-scale multimedia databases has become an important research topic. Cross-modal retrieval methods based on hashing assume that there is a latent space shared by multimodal features. To model the relationship among heterogeneous data, most existing methods embed the data into a joint abstraction space by linear projections. However, these approaches are sensitive to noise in the data and are unable to make use of unlabeled data and multi-modal data with missing values in real-world applications. To address these challenges, we proposed a novel multimodal deep-learning-based hash (MDLH) algorithm. In particular, MDLH uses a deep neural network to encode heterogeneous features into a compact common representation and learns the hash functions based on the common representation. The parameters of the whole model are fine-tuned in a supervised training stage. Experiments on two standard datasets show that the method achieves more effective results than other methods in cross-modal retrieval.
BibTeX
@article{qu2017novel,
  author = {Wen Qu and Daling Wang and Shi Feng and Yifei Zhang and Ge Yu},
  title = {{A novel cross-modal hashing algorithm based on multimodal deep learning}},
  journal = {Science China Information Sciences},
  year = {2017},
  doi = {10.1007/s11432-015-0902-2},
}

Loading related content...

Figure