Multimodal Representation with Embedded Visual Guiding Objects for Named Entity Recognition in Social Media Posts

Zhiwei Wu, Changmeng Zheng, Yi Cai, Junying Chen, Ho-fung Leung, Qing Li

Abstract

Visual contexts often help to recognize named entities more precisely in short texts such as tweets or snapchat. For example, one can identify "Charlie'' as a name of a dog according to the user posts. Previous works on multimodal named entity recognition ignore the corresponding relations of visual objects and entities. Visual objects are considered as fine-grained image representations. For a sentence with multiple entity types, objects of the relevant image can be utilized to capture different entity information. In this paper, we propose a neural network which combines object-level image information and character-level text information to predict entities. Vision and language are bridged by leveraging object labels as embeddings, and a dense co-attention mechanism is introduced for fine-grained interactions. Experimental results in Twitter dataset demonstrate that our method outperforms the state-of-the-art methods.
BibTeX
@inproceedings{wu2020multimodal,
  author = {Zhiwei Wu and Changmeng Zheng and Yi Cai and Junying Chen and Ho-fung Leung and Qing Li},
  title = {{Multimodal Representation with Embedded Visual Guiding Objects for Named Entity Recognition in Social Media Posts}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413650},
}

Loading related content...

Figure