Cross-Modal Learning to Rank via Latent Joint Representation

Fei Wu, Xinyang Jiang, Xi Li, Siliang Tang, Weiming Lu, Zhongfei Zhang, Yueting Zhuang

Abstract

Cross-modal ranking is a research topic that is imperative to many applications involving multimodal data. Discovering a joint representation for multimodal data and learning a ranking function are essential in order to boost the cross-media retrieval (i.e., image-query-text or text-query-image). In this paper, we propose an approach to discover the latent joint representation of pairs of multimodal data (e.g., pairs of an image query and a text document) via a conditional random field and structural learning in a listwise ranking manner. We call this approach cross-modal learning to rank via latent joint representation (CML <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> R). In CML <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> R, the correlations between multimodal data are captured in terms of their sharing hidden variables (e.g., topics), and a hidden-topic-driven discriminative ranking function is learned in a listwise ranking manner. The experiments show that the proposed approach achieves a good performance in cross-media retrieval and meanwhile has the capability to learn the discriminative representation of multimodal data.
BibTeX
@article{wu2015cross,
  author = {Fei Wu and Xinyang Jiang and Xi Li and Siliang Tang and Weiming Lu and Zhongfei Zhang and Yueting Zhuang},
  title = {{Cross-Modal Learning to Rank via Latent Joint Representation}},
  journal = {IEEE Trans. Image Process.},
  year = {2015},
  url = {https://openreview.net/forum?id=-C0IBoQgeCw},
}

Loading related content...

Figure