Multimodal Representation with Embedded Visual Guiding Objects for Named Entity Recognition in Social Media Posts
Lists
Only you see this
Abstract
Visual contexts often help to recognize named entities more precisely in short texts such as tweets or snapchat. For example, one can identify "Charlie'' as a name of a dog according to the user posts. Previous works on multimodal named entity recognition ignore the corresponding relations of visual objects and entities. Visual objects are considered as fine-grained image representations. For a sentence with multiple entity types, objects of the relevant image can be utilized to capture different entity information. In this paper, we propose a neural network which combines object-level image information and character-level text information to predict entities. Vision and language are bridged by leveraging object labels as embeddings, and a dense co-attention mechanism is introduced for fine-grained interactions. Experimental results in Twitter dataset demonstrate that our method outperforms the state-of-the-art methods.
BibTeX
@inproceedings{wu2020multimodal,
author = {Zhiwei Wu and Changmeng Zheng and Yi Cai and Junying Chen and Ho-fung Leung and Qing Li},
title = {{Multimodal Representation with Embedded Visual Guiding Objects for Named Entity Recognition in Social Media Posts}},
booktitle = {ACM Multimedia},
year = {2020},
doi = {10.1145/3394171.3413650},
}Loading related content...