Doubly Semi-Supervised Multimodal Adversarial Learning for Classification, Generation and Retrieval

Changde Du, Changying Du, Huiguang He

Abstract

Learning over incomplete multi-modality data is a challenging problem with strong practical applications. Most existing multi-modal data imputation approaches have two limitations: (1) they are unable to accurately control the semantics of imputed modalities; and (2) without a shared low-dimensional latent space, they do not scale well with multiple modalities. To overcome the limitations, we propose a novel doubly semi-supervised multi-modal learning framework (DSML) with a modality-shared latent space and modality-specific generators, encoders and classifiers. We design novel softmax-based discriminators to train all modules adversarially. As a unified framework, DSML can be applied in multi-modal semi-supervised classification, missing modality imputation and fast cross-modality retrieval tasks simultaneously. Experiments on multiple datasets demonstrate its advantages.
BibTeX
@inproceedings{du2019doubly,
  author = {Changde Du and Changying Du and Huiguang He},
  title = {{Doubly Semi-Supervised Multimodal Adversarial Learning for Classification, Generation and Retrieval}},
  booktitle = {ICME},
  year = {2019},
  doi = {10.1109/icme.2019.00011},
}

Loading related content...

Figure