Turbo your multi-modal classification with contrastive learning

Zhiyu Zhang, Da Liu, Shengqiang Liu, Anna Wang, Jie Gao, Yali Li

Abstract

Contrastive learning has become one of the most impressive approaches for multi-modal representation learning. However, previous multi-modal works mainly focused on cross-modal understanding, ignoring in-modal contrastive learning, which limits the representation of each modality. In this paper, we propose a novel contrastive learning strategy, called $Turbo$, to promote multi-modal understanding by joint in-modal and cross-modal contrastive learning. Specifically, multi-modal data pairs are sent through the forward pass twice with different hidden dropout masks to get two different representations for each modality. With these representations, we obtain multiple in-modal and cross-modal contrastive objectives for training. Finally, we combine the self-supervised Turbo with the supervised multi-modal classification and demonstrate its effectiveness on two audio-text classification tasks, where the state-of-the-art performance is achieved on a speech emotion recognition benchmark dataset.
BibTeX
@misc{zhang2024turbo,
  author = {Zhiyu Zhang and Da Liu and Shengqiang Liu and Anna Wang and Jie Gao and Yali Li},
  title = {{Turbo your multi-modal classification with contrastive learning}},
  year = {2024},
  eprint = {2409.09282},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.09282},
}

Loading related content...

Figure