Multimodal Affective Computing With Dense Fusion Transformer for Inter- and Intra-Modality Interactions

Huan Deng, Zhenguo Yang, Tianyong Hao, Qing Li, Wenyin Liu

Abstract

This paper proposes a dense fusion transformer (DFT) framework to integrate textual, acoustic, and visual information for multimodal affective computing. DFT exploits a modality-shared transformer (MT) module to extract the modality-shared features by modelling unimodal, bimodal, and trimodal interactions jointly. MT constructs a series of dense fusion blocks to fuse utterance-level sequential features of the multiple modalities from the perspectives of low-level and high-level semantics. In particular, MT adopts local and global transformers to learn modality-shared representations by modelling inter- and intra-modality interactions. Furthermore, we devise a modality-specific representation (MR) module with a soft orthogonality constraint to penalize the distance between modality-specific and modality-shared representations, which are fused by a transformer to make affective predictions. Extensive experiments conducted on five public benchmark datasets show that DFT outperforms the state-of-the-art baselines.
BibTeX
@article{deng2023multimodal,
  author = {Huan Deng and Zhenguo Yang and Tianyong Hao and Qing Li and Wenyin Liu},
  title = {{Multimodal Affective Computing With Dense Fusion Transformer for Inter- and Intra-Modality Interactions}},
  journal = {IEEE Trans. Multimedia},
  year = {2023},
  doi = {10.1109/tmm.2022.3211197},
}

Loading related content...

Figure