A Unified Framework for Multimodal Domain Adaptation
Lists
Only you see this
Abstract
Domain adaptation aims to train a model on labeled data from a source domain while minimizing test error on a target domain. Most of existing domain adaptation methods only focus on reducing domain shift of single-modal data. In this paper, we consider a new problem of multimodal domain adaptation and propose a unified framework to solve it. The proposed multimodal domain adaptation neural networks(MDANN) consist of three important modules. (1) A covariant multimodal attention is designed to learn a common feature representation for multiple modalities. (2) A fusion module adaptively fuses attended features of different modalities. (3) Hybrid domain constraints are proposed to comprehensively learn domain-invariant features by constraining single modal features, fused features, and attention scores. Through jointly attending and fusing under an adversarial objective, the most discriminative and domain-adaptive parts of the features are adaptively fused together. Extensive experimental results on two real-world cross-domain applications (emotion recognition and cross-media retrieval) demonstrate the effectiveness of the proposed method.
BibTeX
@inproceedings{qi2018unified,
author = {Fan Qi and Xiaoshan Yang and Changsheng Xu},
title = {{A Unified Framework for Multimodal Domain Adaptation}},
booktitle = {ACM Multimedia},
year = {2018},
doi = {10.1145/3240508.3240633},
}Loading related content...