Multimodal Affective Computing With Dense Fusion Transformer for Inter- and Intra-Modality Interactions
Lists
Only you see this
Abstract
This paper proposes a dense fusion transformer (DFT) framework to integrate textual, acoustic, and visual information for multimodal affective computing. DFT exploits a modality-shared transformer (MT) module to extract the modality-shared features by modelling unimodal, bimodal, and trimodal interactions jointly. MT constructs a series of dense fusion blocks to fuse utterance-level sequential features of the multiple modalities from the perspectives of low-level and high-level semantics. In particular, MT adopts local and global transformers to learn modality-shared representations by modelling inter- and intra-modality interactions. Furthermore, we devise a modality-specific representation (MR) module with a soft orthogonality constraint to penalize the distance between modality-specific and modality-shared representations, which are fused by a transformer to make affective predictions. Extensive experiments conducted on five public benchmark datasets show that DFT outperforms the state-of-the-art baselines.
BibTeX
@article{deng2023multimodal,
author = {Huan Deng and Zhenguo Yang and Tianyong Hao and Qing Li and Wenyin Liu},
title = {{Multimodal Affective Computing With Dense Fusion Transformer for Inter- and Intra-Modality Interactions}},
journal = {IEEE Trans. Multimedia},
year = {2023},
doi = {10.1109/tmm.2022.3211197},
}Loading related content...