Learning Cross-Modal Aligned Representation With Graph Embedding

Youcai Zhang, Jiayan Cao, Xiaodong GuIEEE Access 2018 · OpenReview

Abstract

The main task of cross-modal analysis is to learn discriminative representation shared across different modalities. In order to pursue aligned representation, conventional approaches tend to construct and optimize a linear projection or train a complex architecture of deep layers, yet it is difficult to compromise between accuracy and efficiency on modeling multimodal data. This paper proposes a novel graph-embedding learning framework implemented by neural networks. The learned embedding directly approximates the cross-modal aligned representation to perform cross-modal retrieval and image classification combining text information. Proposed framework extracts learned representation from a graph model and, simultaneously, trains a classifier under semi-supervised settings. For optimization, unlike previous methods based on the graph Laplacian regularization, a sampling strategy is adopted to generate training pairs to fully explore the inter-modal and intra-modal similarity relationship. Experimental results on various datasets show that the proposed framework outperforms other state-of-the-art methods on crossmodal retrieval. The framework also demonstrates convincing improvements on the new issue of image classification combining text information on Wiki dataset.
BibTeX
@inproceedings{zhang2018learning,
  author = {Youcai Zhang and Jiayan Cao and Xiaodong Gu},
  title = {{Learning Cross-Modal Aligned Representation With Graph Embedding}},
  booktitle = {IEEE Access},
  year = {2018},
  url = {https://openreview.net/forum?id=GW7f0Xbijo},
}

Loading related content...

Figure