Multimodal Transformer Networks with Latent Interaction for Audio-Visual Event Localization

Yixuan He, Xing Xu, Xin Liu, Weihua Ou, Huimin LuICME 2021 · OpenReview

Abstract

The task of audio-visual event localization (AVEL) aims to localize a visible and audible event in a video. Previous methods first divide a video into segments and then fuse visual and acoustic features at the segment level via a co-attention mechanism. However, existing methods mostly model relations between individual visual and audio segments in a limitedly short period, which may not cover a longer video duration for better high-level event information modeling. In this paper, we proposed a novel model termed Multimodal Transformer Network with Latent Interaction (MTNLI) to tackle this problem. The proposed MTNLI model employs a multimodal Transformer structure to learn the cross-modality relationships between latent visual and audio summarizations in long segment sequences, which summarize the visual and audio segments into a small number of latent representations to avoid modeling uninformative individual visual-audio relations. The cross-modality information between the latent summarizations is propagated to fuse valuable information from both modalities, which can effectively handle large temporal inconsistent between vision and audio. Our MTNLI method achieves state-of-the-art performance on the benchmark AVE (Audio-Visual Event) dataset for the event localization task.
BibTeX
@inproceedings{he2021multimodal,
  author = {Yixuan He and Xing Xu and Xin Liu and Weihua Ou and Huimin Lu},
  title = {{Multimodal Transformer Networks with Latent Interaction for Audio-Visual Event Localization}},
  booktitle = {ICME},
  year = {2021},
  url = {https://openreview.net/forum?id=u1GYxBEKFg-},
}

Loading related content...

Figure