Turbo your multi-modal classification with contrastive learning
Lists
Only you see this
Abstract
Contrastive learning has become one of the most impressive approaches for multi-modal representation learning. However, previous multi-modal works mainly focused on cross-modal understanding, ignoring in-modal contrastive learning, which limits the representation of each modality. In this paper, we propose a novel contrastive learning strategy, called $Turbo$, to promote multi-modal understanding by joint in-modal and cross-modal contrastive learning. Specifically, multi-modal data pairs are sent through the forward pass twice with different hidden dropout masks to get two different representations for each modality. With these representations, we obtain multiple in-modal and cross-modal contrastive objectives for training. Finally, we combine the self-supervised Turbo with the supervised multi-modal classification and demonstrate its effectiveness on two audio-text classification tasks, where the state-of-the-art performance is achieved on a speech emotion recognition benchmark dataset.
BibTeX
@misc{zhang2024turbo,
author = {Zhiyu Zhang and Da Liu and Shengqiang Liu and Anna Wang and Jie Gao and Yali Li},
title = {{Turbo your multi-modal classification with contrastive learning}},
year = {2024},
eprint = {2409.09282},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2409.09282},
}Loading related content...