Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge

Jinyuan Li, Han Li, Zhuo Pan, Gang Pan, Wang, Jiahao, Zhang, Wenkun, Pan, Gang

Abstract

Multimodal Named Entity Recognition (MNER) on social media aims to enhance textual entity prediction by incorporating image-based clues. Existing studies mainly focus on maximizing the utilization of pertinent image information or incorporating external knowledge from explicit knowledge bases. However, these methods either neglect the necessity of providing the model with external knowledge, or encounter issues of high redundancy in the retrieved knowledge. In this paper, we present PGIM — a two-stage framework that aims to leverage ChatGPT as an implicit knowledge base and enable it to heuristically generate auxiliary knowledge for more efficient entity prediction. Specifically, PGIM contains a Multimodal Similar Example Awareness module that selects suitable examples from a small number of predefined artificial samples. These examples are then integrated into a formatted prompt template tailored to the MNER and guide ChatGPT to generate auxiliary refined knowledge. Finally, the acquired knowledge is integrated with the original text and fed into a downstream model for further processing. Extensive experiments show that PGIM outperforms state-of-the-art methods on two classic MNER datasets and exhibits a stronger robustness and generalization capability.

Introduction

Named Entity Recognition (NER) on social media platforms like Twitter is notoriously difficult due to brief, informal posts filled with slang and abbreviations. Multimodal NER (MNER) attempts to mitigate this by using images for context. However, current approaches struggle with the "modality gap"—aligning raw image features with text—and "knowledge redundancy," where external retrievals (e.g., from Wikipedia) introduce noisy or irrelevant information.

PGIM (Prompting ChatGPT In MNER) addresses these hurdles by treating ChatGPT as an "implicit knowledge base." Instead of raw data dumps, PGIM prompts the LLM to generate refined, heuristic reasoning that provides high-quality context for downstream models.

The PGIM Framework

The PGIM framework operates in two distinct stages to bridge the gap between multimodal inputs and precise entity labeling.

Stage 1: Auxiliary Refined Knowledge Generation

First, images are converted into descriptive captions using BLIP-2. To guide ChatGPT effectively, the Multimodal Similar Example Awareness (MSEA) module identifies the top-N most relevant examples from a small set of manually annotated "predefined artificial samples." These samples include both entities and the reasoning behind them. ChatGPT is then prompted with these examples to generate an "auxiliary explanation"—for instance, explaining that "CP3" is a common nickname for NBA player Chris Paul based on the visual context.

The PGIM architecture overview, illustrating the two-stage process from multimodal input to ChatGPT-driven knowledge refinement and final entity prediction.

Stage 2: Entity Prediction

The original text is concatenated with ChatGPT’s generated reasoning and fed into a Transformer-based encoder (XLM-RoBERTa). A Conditional Random Field (CRF) layer then predicts the final entity labels, ensuring the output adheres to standard BIO tagging logic.

Experimental Results

PGIM was evaluated on the benchmark Twitter-2015 and Twitter-2017 datasets. The framework achieved state-of-the-art (SOTA) performance across the board, significantly outperforming previous multimodal models.

Performance comparison of PGIM against state-of-the-art text-only and multimodal NER methods on the Twitter-2015 and Twitter-2017 datasets.

As shown in the results, PGIM achieved an F1-score of 79.33 on Twitter-2015 and 91.43 on Twitter-2017. Notably, PGIM demonstrated superior generalization in cross-dataset tests and proved more resource-efficient than retrieval-augmented models like MoRe, as ChatGPT's refined outputs are significantly more concise than Wikipedia entries.

Discussion and Conclusion

A primary insight of this research is that ChatGPT’s reasoning is more valuable than its raw predictions for sequence labeling tasks. By providing the logic behind an entity (e.g., why a person is likely an athlete based on their uniform), the final classifier becomes more robust and less susceptible to noise.

While the current framework relies on image captioning—which can lose some visual nuance—and external API calls, it establishes a powerful new paradigm for MNER. Future work will likely explore native multimodal LLMs to eliminate the captioning bottleneck, allowing the model to "see" the image directly while generating heuristic knowledge.

Summary updated

Abstract

Multimodal Named Entity Recognition (MNER) on social media aims to enhance textual entity prediction by incorporating image-based clues. Existing studies mainly focus on maximizing the utilization of pertinent image information or incorporating external knowledge from explicit knowledge bases. However, these methods either neglect the necessity of providing the model with external knowledge, or encounter issues of high redundancy in the retrieved knowledge. In this paper, we present PGIM -- a two-stage framework that aims to leverage ChatGPT as an implicit knowledge base and enable it to heuristically generate auxiliary knowledge for more efficient entity prediction. Specifically, PGIM contains a Multimodal Similar Example Awareness module that selects suitable examples from a small number of predefined artificial samples. These examples are then integrated into a formatted prompt template tailored to the MNER and guide ChatGPT to generate auxiliary refined knowledge. Finally, the acquired knowledge is integrated with the original text and fed into a downstream model for further processing. Extensive experiments show that PGIM outperforms state-of-the-art methods on two classic MNER datasets and exhibits a stronger robustness and generalization capability.
BibTeX
@inproceedings{li2023prompting,
  author = {Jinyuan Li and Han Li and Zhuo Pan and Gang Pan and Wang, Jiahao and Zhang, Wenkun and Pan, Gang},
  title = {{Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.184},
}
Figure