Incorporating Object-Level Visual Context for Multimodal Fine-Grained Entity Typing

Ying Zhang, Wenbo Fan, Kehui Song, Yu Zhao, Xuhui Sui, Xiaojie Yuan

Introduction

Fine-Grained Entity Typing (FGET) is a fundamental task in Natural Language Processing (NLP) that involves assigning specific, hierarchical labels to entity mentions within a text. While identifying a mention like "Lionel Messi" as a Person is straightforward, classifying him as an Athlete requires nuanced context. A significant challenge in traditional FGET is that textual context is often sparse or ambiguous. For instance, a sentence stating "Lionel Messi won the championship" does not explicitly define his role; he could be a coach or an owner. However, visual cues from an accompanying image—such as a soccer ball or a jersey—can immediately disambiguate the entity.

To address this, we introduce Multimodal Fine-Grained Entity Typing (MFGET) and propose the Multimodal Object-Level Visual Context Network (MOVCNet). Unlike previous multimodal approaches that rely on global image features, MOVCNet leverages object-level visual context to provide precise disambiguation.

The MOVCNet Architecture

MOVCNet is designed to integrate textual information with specific visual objects detected within an image. The architecture consists of three primary components:

  1. Representation Layer: Textual mentions are encoded using a standard BERT model. For visual representation, instead of global features, we utilize VinVL, a pre-trained object detector, to extract features from specific objects (e.g., microphones, flags, sports equipment) that serve as strong indicators of an entity's type.
  2. Fusion Layer (Text-Guided Attention): To filter out irrelevant visual noise, a Multi-Head Attention mechanism is employed. The textual embedding serves as the Query, while the detected objects act as Keys and Values. This allows the model to focus exclusively on objects relevant to the entity mentioned in the text.
  3. Hybrid Classification: A dual-classifier setup handles the complex label hierarchy, predicting types independently while also capturing latent correlations between labels using Singular Value Decomposition (SVD).
The architecture of MOVCNet, illustrating the flow from Representation to Hybrid Classification.

The MFIGER Dataset

To evaluate this approach, we developed the MFIGER benchmark. This dataset was constructed by augmenting the existing FIGER dataset, mapping Wikipedia titles to Wikidata IDs to retrieve nearly one million image-text pairs. MFIGER covers 102 fine-grained types organized into a two-level hierarchy, providing a robust platform for testing multimodal entity typing.

Experimental Results

MOVCNet achieves state-of-the-art performance, significantly outperforming both text-only and previous multimodal baselines like UMT. The model reached a Strict Accuracy of 90.99%, a notable improvement over UMT's 88.56%.

Performance comparison of MOVCNet against baselines across eight different categories (C1-C8), showing consistent superiority.

The most significant gains were observed in "Fine" label classification, where MOVCNet saw a 2.24% boost in strict accuracy. Ablation studies confirmed that the use of object-level features (VinVL) over global features (VGG16) is the primary driver of this performance increase, as local objects provide the necessary "complementary semantics" to resolve textual ambiguity.

Conclusion

By shifting the focus from global image scenes to specific, text-relevant objects, MOVCNet effectively addresses the limitations of textual context in fine-grained entity typing. Future work will explore end-to-end training of the visual extractor and more sophisticated text-vision interaction mechanisms to further refine classification accuracy in real-time systems.

Summary updated

Abstract

Fine-grained entity typing (FGET) aims to assign appropriate fine-grained types to entity mentions within their context, which is an important foundational task in natural language processing. Previous approaches for FGET only utilized textual context information. However, in the form of short text, the contextual semantic information is often insufficient for FGET. In many real-world scenarios, text is often accompanied by images, and the visual context is valuable for FGET. To this end, we firstly propose a new task called multimodal fine-grained entity typing (MFGET). Then we construct a large-scale dataset for multimodal fine-grained entity typing called MFIGER based on FIGER. To fully leverage both textual and visual information, we propose a novel Multimodal Object-Level Visual Context Network (MOVCNet). MOVCNet can capture fine-grained semantic information by detecting objects in images, and effectively merge both textual and visual context. Experimental results demonstrate that our approach achieves superior classification performance compared to previous text-based approaches.
BibTeX
@inproceedings{zhang2023incorporating,
  author = {Ying Zhang and Wenbo Fan and Kehui Song and Yu Zhao and Xuhui Sui and Xiaojie Yuan},
  title = {{Incorporating Object-Level Visual Context for Multimodal Fine-Grained Entity Typing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.1027},
}
Figure