Note: This paper was formally retracted in October 2025. While the AAAI Press noted that the technical approach and results were novel, portions of the related work section were found to have excessive textual similarity to existing literature. This summary focuses on the technical contributions described in the original text.
Abstract
Multi-modal named entity recognition (MNER) aims to discover named entities in free text and classify them into pre-defined types with images. However, dominant MNER models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities, which have the potential to refine multi-modal representation learning. To deal with this issue, we propose a unified multi-modal graph fusion (UMGF) approach for MNER. Specifically, we first represent the input sentence and image using a unified multi-modal graph, which captures various semantic relationships between multi-modal semantic units (words and visual objects). Then, we stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions to learn node representations. Finally, we achieve an attention-based multi-modal representation for each word and perform entity labeling with a CRF decoder. Experimentation on the two benchmark datasets demonstrates the superiority of our MNER model.
Introduction
Named Entity Recognition (NER) is a fundamental task in natural language processing, identifying and categorizing entities such as people, locations, and organizations. In the context of social media, text is frequently accompanied by images that provide essential context for disambiguation. This task, known as Multi-modal NER (MNER), faces a significant challenge in fusion: effectively combining sequential text with visual data.
Traditional models often rely on "global" image features or simple grid-based representations. These methods can introduce noise, as large portions of an image (like the sky or background tourists) may be irrelevant to the specific entities mentioned in the text. The authors argue that effective MNER requires "targeted visual guidance"—explicitly linking specific words to the corresponding visual objects within an image.
The UMGF Framework
The core of the proposed approach is the Unified Multi-modal Graph Fusion (UMGF) framework. This model treats MNER as a graph modeling task, bridging the semantic gap between modalities by representing both text and images as nodes in a shared space.
1. Node and Edge Construction
The graph is constructed using two types of nodes:
- Textual Nodes: Each word in the sentence is represented as a node.
- Visual Nodes: The model extracts noun phrases from the text and uses a visual grounding toolkit to identify corresponding regions (objects) in the image. These regions, along with "general" category nodes (e.g., "person"), form the visual nodes.
Edges are then established to define relationships:
- Intra-modal Edges: Connect all textual nodes to each other and all visual nodes to each other.
- Inter-modal Edges: These are the "targeted" links that connect specific words (like "Brandenburg Gate") to their corresponding visual objects detected in the image.
2. Stacked Fusion and Decoding
The model employs multiple graph-based fusion layers. Each layer performs Intra-modal Fusion via self-attention, followed by Inter-modal Fusion using a gating mechanism. This gated interaction allows the model to dynamically weigh the importance of visual information for each word, mitigating the impact of misleading or irrelevant images. Finally, a Conditional Random Field (CRF) decoder predicts entity labels based on the enriched representations.
Experimental Results
The UMGF model was evaluated on three benchmark datasets: Twitter-2015, Twitter-2017, and a "cleaned" version, Twitter-2017c.
- State-of-the-Art Performance: UMGF achieved an F1 score of 74.85% on Twitter-2015, surpassing the previous best (UMT) of 73.41%. On Twitter-2017, it reached 85.51%.
- Ablation Study: To verify the importance of targeted guidance, the authors tested a version of the model using simple image segmentation instead of word-to-object mapping (UMGF w/o Tar.). The F1 score on Twitter-2015 dropped significantly from 74.85% to 71.21%, confirming that specific visual grounding is the primary driver of performance.
Discussion and Limitations
The primary insight of this work is that not all pixels are equally relevant. By focusing on visual evidence that directly confirms an entity's type, the model effectively ignores background noise. However, the model's performance is inherently tied to the accuracy of the initial noun phrase extraction and visual grounding tools. If the object detector fails to locate the relevant entity in the image, the benefits of targeted guidance are lost.
While the paper was retracted due to textual similarities in the related work section, the technical contribution of the graph-based fusion mechanism remains a notable approach for multi-modal tasks in social media environments.
Summary updated