MAVEN: A Memory Augmented Recurrent Approach for Multimodal Fusion

Md Mofijul Islam, Mohammad Samin Yasar, Tariq IqbalIEEE Trans. Multimedia 2023 · OpenReview

Abstract

Multisensory systems provide complementary information that aids many machine learning approaches in perceiving the environment comprehensively. These systems consist of heterogeneous modalities, which have disparate characteristics and feature distributions. Thus, extracting, aligning, and fusing complementary representations from heterogeneous modalities (e.g., visual, skeleton, and physical sensors) remains challenging. To address these challenges, we have used the insights from several neuroscience studies of animal multisensory systems to develop MAVEN, a memory-augmented recurrent approach for multimodal fusion. MAVEN generates unimodal memory banks comprised of spatial-temporal features and uses our proposed recurrent representation alignment approach to align and refine unimodal representations iteratively. MAVEN then utilizes a multimodal variational attention-based fusion approach to produce a robust multimodal representation from the aligned unimodal features. Our extensive experimental evaluations on three multimodal datasets suggest that MAVEN outperforms state-of-the-art multimodal learning approaches in the challenging human activity recognition task across all evaluation conditions (cross-subject, leave-one-subject-out, and cross-session). Additionally, our extensive ablation studies suggest that MAVEN significantly outperforms the feed-forward fusion-based learning models <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(p&lt; 0.05)$</tex-math></inline-formula> . Finally, the robust performance of MAVEN in extracting complementary multimodal representation from occluded and noisy data suggests its applicability on real-world datasets.
BibTeX
@article{islam2023maven,
  author = {Md Mofijul Islam and Mohammad Samin Yasar and Tariq Iqbal},
  title = {{MAVEN: A Memory Augmented Recurrent Approach for Multimodal Fusion}},
  journal = {IEEE Trans. Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=tPi3o9dDbr},
}

Loading related content...

Figure