Good Visual Guidance Make A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction

Xiang Chen, Ningyu Zhang, Lei Li, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, Huajun Chen

Introduction

In social media contexts, text is frequently accompanied by visual content. Multimodal Named Entity Recognition (MNER) and Multimodal Relation Extraction (MRE) leverage these images to disambiguate textual information—for instance, determining whether "Apple" refers to the fruit or the corporation. However, a significant hurdle in current research is error sensitivity. Many existing models are easily distracted by irrelevant images (e.g., a celebrity photo attached to a news report about a car crash), leading to incorrect extractions. The challenge lies in effectively fusing disparate data types (pixels and text) while ensuring the model can selectively ignore visual noise.

HVPNeT demonstrates superior robustness to irrelevant images and better utilization of relevant visual cues compared to existing models.

Methodology: The HVPNeT Framework

To address these challenges, the authors introduce the Hierarchical Visual Prefix fusion NeTwork (HVPNeT). The architecture is built upon three core innovations designed to provide robust visual guidance.

1. Visual Prefix-guided Fusion

Inspired by "prompt tuning" in natural language processing, HVPNeT treats visual features as a "prefix" (a set of virtual tokens) prepended to the text sequence. These prefixes are integrated into every self-attention layer of the BERT backbone. This "pluggable" approach allows for deep multimodal interaction without requiring fundamental changes to the underlying Transformer architecture.

2. Pyramidal Visual Features

Rather than relying solely on the final output of a vision model (e.g., ResNet), HVPNeT extracts features from multiple stages or blocks of the CNN. This captures a hierarchy of visual information, ranging from low-level textures and shapes in early blocks to high-level object semantics in later stages.

3. Dynamic Gated Aggregation

Recognizing that different visual scales are relevant to different stages of text processing, the model employs a dynamic gate. This gate adaptively weights and aggregates multi-scale features for each specific Transformer layer, effectively "routing" the most appropriate visual guidance to the corresponding level of the text extraction process.

The architecture of HVPNeT featuring a ResNet encoder for multi-scaled features, dynamic gates for aggregation, and a hierarchical visual prefix for Transformer-based fusion.

Experimental Evaluation

The model was rigorously tested on three benchmark datasets: Twitter-2015 and Twitter-2017 for MNER, and MNRE for MRE. The MNRE task involves identifying complex relationships between entities, such as MemberOf or LocatedIn, using both textual and visual evidence.

Key Results

  • State-of-the-Art Performance: HVPNeT established new performance benchmarks across all datasets. Most notably, on the MNRE task, it achieved an F1 score of 81.85, representing a substantial improvement over the previous state-of-the-art (MEGA) at 66.41.
  • Data Efficiency: The model demonstrates remarkable low-resource stability. When trained on just 5% of the available data, HVPNeT outperformed several competitive baselines that had access to significantly more training information.
  • Cross-Task Robustness: In cross-task scenarios, HVPNeT exhibited minimal performance degradation compared to its predecessors, suggesting the learning of more generalized multimodal representations rather than dataset-specific biases.

Discussion and Key Insights

A primary takeaway from HVPNeT is the effectiveness of images as prompts. By treating visual data as guidance rather than a mandatory part of the sentence structure, the model preserves textual integrity while benefiting from visual context. Furthermore, the hierarchical approach aligns with the internal logic of Transformers: low-level visual features assist early layers (handling syntax), while high-level semantics assist later layers (handling deep meaning).

This "soft" fusion method makes HVPNeT significantly more robust to irrelevant images. When faced with visual noise, the model often defaults to correct text-only predictions, whereas other models are frequently misled.

Limitations and Future Work

Currently, HVPNeT focuses on using vision to aid text extraction but does not explicitly improve visual grounding (the precise mapping of words to image regions). Future research may explore reversing the information flow to enhance visual representations using text, scaling the prefix approach to massive multi-task pre-training, and extending the framework to a wider variety of backbone architectures.

Summary updated

BibTeX
@inproceedings{chen2022good,
  author = {Xiang Chen and Ningyu Zhang and Lei Li and Yunzhi Yao and Shumin Deng and Chuanqi Tan and Fei Huang and Luo Si and Huajun Chen},
  title = {{Good Visual Guidance Make A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.121},
}
Figure