Multi-Modal Language Analysis with Hierarchical Interaction-Level and Selection-Level Attentions
Lists
Only you see this
Abstract
As an emerging research area in natural language processing, multi-modal human language analysis spans language, vision and audio modalities. Understanding multi-modal language requires not only the modeling of independent dynamics within each modality (intra-modal dynamics), but also more importantly interactive dynamics among different modalities (inter-modal dynamics). In this paper, we propose a hierarchical approach to multi-modal language analysis with two levels of attention mechanism, namely interaction-level, which captures the intra-modal and inter-modal dynamics across different modalities with multiple types of attention, and selection-level attention, which selects the effective representations for final prediction by calculating the importance of each vector obtained from interaction-level. Empirical evaluation demonstrates the effectiveness of our proposed approach to multi-modal sentiment classification, sentiment regression and emotion recognition.
BibTeX
@inproceedings{zhang2019multi,
author = {Dong Zhang and Liangqing Wu and Shoushan Li and Qiaoming Zhu and Guodong Zhou},
title = {{Multi-Modal Language Analysis with Hierarchical Interaction-Level and Selection-Level Attentions}},
booktitle = {ICME},
year = {2019},
url = {https://openreview.net/forum?id=bBazIf35eV},
}Loading related content...