Token-disentangling Mutual Transformer for multimodal emotion recognition

Guanghao Yin, Yuanyuan Liu, Tengfei Liu, Haoyu Zhang, Fang Fang, Chang Tang, Liangxiao Jiang

Abstract

Highlights•We propose a novel and robust Token-disentangling Mutual Transformer (TMT) to achieve accurate multimodal emotion recognition.•We propose a simple and easy-to-implement multimodal emotion Token disentanglement module to disentangle the inter-modality consistency features and intra-modality heterogeneity features.•We introduce a Token separation encoder to help the Transformer separate four groups of features without taking additional parameters.•We devise a Token mutual Transformer to fully interact and integrate the emotion consistency and heterogeneity information by exploring their mutual contribution in emotion interactions.
BibTeX
@article{yin2024token,
  author = {Guanghao Yin and Yuanyuan Liu and Tengfei Liu and Haoyu Zhang and Fang Fang and Chang Tang and Liangxiao Jiang},
  title = {{Token-disentangling Mutual Transformer for multimodal emotion recognition}},
  journal = {Engineering Applications of AI},
  year = {2024},
  doi = {10.1016/j.engappai.2024.108348},
}

Loading related content...

Figure