Multimodal Named Entity Recognition for Short Social Media Posts

Seungwhan Moon, Leonardo Neves, Vitor Carvalho

TL;DR

This paper introduces Multimodal Named Entity Recognition (MNER), a framework designed to identify entities in short, noisy social media posts by leveraging both text and accompanying images. By employing a novel modality attention mechanism, the model learns to dynamically weight word, character, and visual features, significantly outperforming traditional text-only models in resolving ambiguities and handling informal language.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a foundational NLP task, but it struggles with the chaotic nature of social media. The authors identify two primary hurdles:

  1. Limited Textual Context: Posts on platforms like Snapchat or Twitter are often extremely short. Without enough surrounding text, polysemous words (words with multiple meanings) are impossible to disambiguate. For example, does "monopoly" refer to the board game (an entity) or an economic concept?
  2. Noisy Text: Social media is rife with typos, slang, and emerging trends (e.g., "Marshmelloooo" for the DJ Marshmello). These "out-of-vocabulary" (OOV) tokens break standard word-embedding-based models, which often default to a generic "unknown" (UNK) representation, losing critical information.
Visual context helps resolve polysemy (a) and modality attention handles noisy tokens (b)

Methods

The authors build upon the standard Bi-LSTM-CRF architecture but augment it with two major innovations:

  • Visual Feature Extraction: They use a pre-trained InceptionNet (GoogLeNet) to extract high-level visual descriptors from images. These features provide a "visual context" that acts as a proxy for the missing textual context.
  • Modality Attention Module: Instead of simply concatenating word, character, and visual features—a "naive" approach that treats all inputs as equally important—they introduce an attention mechanism. This module calculates a weighted average of the three modalities for every single token.

If a word is a typo (like "WaRriOoOrs"), the model can "turn down" the noisy word embedding and "turn up" the character-level features or the visual context to make a correct prediction.

The MNER architecture featuring the modality attention module merging image, word, and character features.

Datasets

The researchers introduced a new benchmark:

  • SnapCaptions: A dataset of 10,000 Snapchat image-caption pairs manually annotated for four entity types (PER, LOC, ORG, MISC). These are real-world, crowd-sourced "Stories" that are notoriously difficult to process due to their brevity (average 5.8 words per caption).
  • Baselines: They compared their model against state-of-the-art text-only models (e.g., Lample et al., 2016; Ma and Hovy, 2016) and multimodal versions using simple feature concatenation.

Key Insights and Ideas

  • Images as Disambiguators: The "aha moment" is seeing how an image of a board game immediately tells the model that "monopoly" is a MISC entity, even when the text "monopoly is da best" provides zero clues.
  • Attention vs. Concatenation: Naive concatenation is vulnerable to noise. If one modality is "garbage" (like a random vector for an unknown word), it pollutes the entire feature vector. Modality attention acts as a smart filter, effectively "switching off" the broken modality for that specific token.
  • Robustness to Vocabulary Size: The authors found that modality attention makes the model more robust to smaller word-embedding vocabularies. By focusing more on characters and images when words are missing, the model maintains high performance even when the word-embedding matrix is artificially restricted.

Key Results

The multimodal approach yielded significant gains across the board on the SnapCaptions dataset:

  • Entity Type Recognition: The proposed multimodal model with modality attention achieved an F1 score of 52.4%, compared to 48.8% for the best text-only baseline.
  • Visual Impact: Adding visual context (W+C+V) consistently outperformed text-only (W+C) configurations, proving that images are not just "extra" data but essential context for short-form content.
  • Qualitative Success: The model successfully identified trending entities like "Marshmello" and "Duke Dumont" by attending to visual cues of concert scenes when the text was insufficient.
Figure 3: Visualization of modality attention (a) successful cases and (b) unsuccessful ones from SnapCaptions test data. For each decoding step of a token (column), the modality attention module amplifies the most relevant modality (darker) while attenuating irrelevant modalities (lighter). The model makes final predictions based on weighted signals from all modalities. For interpretability, visual tags (label output of InceptionNet) are presented instead of actual feature vectors used. GT: ground-truth, Pred: prediction by our model. Modalities- W: words, C: characters, V: visual.

Limitations and Future Work

  • Misleading Visuals: The model can be "tricked" by irrelevant images. For instance, if a user posts a caption about a "HUUUGE" sale with a picture of a DJ, the model might incorrectly flag "HUUUGE" as a person (PER) because it over-attends to the visual context of a concert.
  • Visual Tag Granularity: The current model uses high-level visual tags from InceptionNet. Future work could explore more granular object detection or spatial attention to link specific parts of an image to specific words in a caption.
  • Dataset Scale: While 10k pairs is a solid start, social media trends move fast. Scaling the dataset to include more diverse and rapidly changing entities remains a challenge.

Summary updated

Abstract

We introduce a new task called Multimodal Named Entity Recognition (MNER) for noisy user-generated data such as tweets or Snapchat captions, which comprise short text with accompanying images. These social media posts often come in inconsistent or incomplete syntax and lexical notations with very limited surrounding textual contexts, bringing significant challenges for NER. To this end, we create a new dataset for MNER called SnapCaptions (Snapchat image-caption pairs submitted to public and crowd-sourced stories with fully annotated named entities). We then build upon the state-of-the-art Bi-LSTM word/character based NER models with 1) a deep image network which incorporates relevant visual context to augment textual information, and 2) a generic modality-attention module which learns to attenuate irrelevant modalities while amplifying the most informative ones to extract contexts from, adaptive to each sample and token. The proposed MNER model with modality attention significantly outperforms the state-of-the-art text-only NER models by successfully leveraging provided visual contexts, opening up potential applications of MNER on myriads of social media platforms.
BibTeX
@inproceedings{moon2018multimodal,
  author = {Seungwhan Moon and Leonardo Neves and Vitor Carvalho},
  title = {{Multimodal Named Entity Recognition for Short Social Media Posts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=HkbRUXZ_ZS},
}
Figure