Multi-Modal Language Analysis with Hierarchical Interaction-Level and Selection-Level Attentions

Dong Zhang, Liangqing Wu, Shoushan Li, Qiaoming Zhu, Guodong ZhouICME 2019 · OpenReview

Abstract

As an emerging research area in natural language processing, multi-modal human language analysis spans language, vision and audio modalities. Understanding multi-modal language requires not only the modeling of independent dynamics within each modality (intra-modal dynamics), but also more importantly interactive dynamics among different modalities (inter-modal dynamics). In this paper, we propose a hierarchical approach to multi-modal language analysis with two levels of attention mechanism, namely interaction-level, which captures the intra-modal and inter-modal dynamics across different modalities with multiple types of attention, and selection-level attention, which selects the effective representations for final prediction by calculating the importance of each vector obtained from interaction-level. Empirical evaluation demonstrates the effectiveness of our proposed approach to multi-modal sentiment classification, sentiment regression and emotion recognition.
BibTeX
@inproceedings{zhang2019multi,
  author = {Dong Zhang and Liangqing Wu and Shoushan Li and Qiaoming Zhu and Guodong Zhou},
  title = {{Multi-Modal Language Analysis with Hierarchical Interaction-Level and Selection-Level Attentions}},
  booktitle = {ICME},
  year = {2019},
  url = {https://openreview.net/forum?id=bBazIf35eV},
}

Loading related content...

Figure