Gated Multimodal Units for Information Fusion
Lists
Only you see this
Abstract
This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.
BibTeX
@inproceedings{arevalo2017gated,
author = {John Arevalo and Thamar Solorio and Manuel Montes-y-Gómez and Fabio A. González},
title = {{Gated Multimodal Units for Information Fusion}},
booktitle = {International Conference on Learning Representations},
year = {2017},
url = {https://openreview.net/forum?id=Hy-2G6ile},
}| Learn to Combine Modalities in Multimodal Deep Learning | arXiv · 2018 |
| Adaptive Fusion Techniques for Multimodal Data | EACL · 2021 |
| MMTM: Multimodal Transfer Module for CNN Fusion | CVPR · 2020 |
| Dynamic Fusion for Multimodal Data | arXiv · 2019 |
| Beyond Simple Fusion: Adaptive Gated Fusion for Robust Multimodal Sentiment Analysis | arXiv · 2025 |
| CentralNet: A Multilayer Approach for Multimodal Fusion | Workshops @ ECCV · 2018 |