Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction

Xiang Chen, Ningyu Zhang, Lei Li, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, Huajun Chen

Abstract

Multimodal Information Extraction (MIE) aims to enhance text-based entity and relation extraction by leveraging associated visual information. However, existing methods often struggle with "noise"—irrelevant images that degrade performance compared to text-only baselines. This paper introduces HVPNeT (Hierarchical Visual Prefix fusion NeTwork), a novel architecture that treats multi-scale visual features as hierarchical prefixes to guide the attention mechanism of a pre-trained language model. By utilizing a dynamic gated aggregation mechanism, HVPNeT effectively filters irrelevant visual noise while capturing multi-level cross-modal alignments. Experimental results demonstrate that HVPNeT achieves state-of-the-art performance on MNER and MRE benchmarks and exhibits superior robustness and data efficiency.

Introduction

In the domain of Information Extraction (IE), the integration of multimodal data—specifically text and images—has become a vital frontier. Tasks such as Multimodal Named Entity Recognition (MNER) and Multimodal Relation Extraction (MRE) rely on the premise that visual context can disambiguate textual references. For instance, an image of a basketball court can clarify that a mention of "LeBron" refers to the professional athlete rather than a different individual.

Despite the potential of multimodal models, current architectures like UMGF or MEGA face a significant hurdle: error sensitivity. In social media contexts, images are frequently only tangentially related or entirely irrelevant to the accompanying text. When faced with such noise, many existing models perform worse than their text-only counterparts. The challenge lies in developing a system that can leverage helpful visual clues without being distracted by irrelevant ones.

Methodology

HVPNeT addresses these challenges by moving away from traditional feature concatenation, instead adopting a "prefix-tuning" inspired approach to multimodal fusion.

1. Visual Prefix-guided Fusion

Rather than simply appending visual vectors to textual embeddings, HVPNeT treats visual features as "pluggable" prefixes. These prefixes are prepended to the keys ($K$) and values ($V$) of every self-attention layer within a pre-trained BERT model. This allows the visual data to function as a prompt, modulating how the textual representation is formed at each layer of the transformer.

2. Hierarchical Pyramidal Features

Linguistic abstraction in models like BERT varies across layers, moving from surface-level syntax to deep semantics. To match this, HVPNeT extracts visual features from multiple blocks of a ResNet backbone, creating a "pyramid" of visual information. This ensures that fine-grained textures and global object shapes are available to the textual encoder at the appropriate stages of processing.

3. Dynamic Gated Aggregation

To manage the interaction between different visual scales and textual layers, the model employs a learnable gating mechanism. This mechanism adaptively weights and aggregates multi-scale features, effectively "routing" the most relevant visual information to the corresponding level of text processing. This gating is crucial for robustness, as it allows the model to suppress irrelevant visual features that do not align with the text.

Experimental Evaluation

The performance of HVPNeT was evaluated on three standard multimodal benchmarks:

  • Twitter-2015 & Twitter-2017: Benchmarks for Multimodal Named Entity Recognition (MNER).
  • MNRE: The primary benchmark for Multimodal Relation Extraction (MRE), consisting of over 9,000 image-text pairs.

The implementation utilized ResNet50 as the visual backbone and BERT-base as the textual encoder.

Key Results

HVPNeT established new state-of-the-art (SOTA) benchmarks across all tested datasets:

  • MNER: On the Twitter-2017 dataset, the model achieved an F1 score of 86.87%, representing a 1.36% improvement over the previous best model (UMGF).
  • MRE: On the MNRE dataset, HVPNeT reached an F1 score of 81.85%, a substantial improvement over the previous SOTA (MEGA at 66.41%).
  • Robustness: Cross-task generalization tests showed that HVPNeT experienced significantly less performance degradation than previous models when moved across different datasets.
  • Efficiency: In low-resource scenarios (using only 5% to 10% of training data), HVPNeT maintained a significant performance lead, suggesting high data efficiency.

Discussion and Insights

The success of HVPNeT stems from two core insights. First, treating images as prefixes rather than input tokens allows the model to treat visual data as context that modulates the text. Second, the alignment of scales—matching low-level visual features with syntactic layers and high-level concepts with semantic layers—provides a more nuanced fusion than flat concatenation. The dynamic gating mechanism serves as a critical filter, solving the long-standing problem of multimodal models being less reliable than text-only models in the presence of noise.

Limitations and Future Work

While HVPNeT represents a significant advancement, it currently focuses on unidirectional guidance (vision guiding text). Future research could explore bidirectional guidance to improve computer vision tasks using textual prompts. Additionally, the model does not currently perform visual grounding (linking entities to specific pixels). Extending this hierarchical prefix approach to large-scale multimodal pre-training remains a promising avenue for future development.

Summary updated

Abstract

Multimodal named entity recognition and relation extraction (MNER and MRE) is a fundamental and crucial branch in information extraction. However, existing approaches for MNER and MRE usually suffer from error sensitivity when irrelevant object images incorporated in texts. To deal with these issues, we propose a novel Hierarchical Visual Prefix fusion NeTwork (HVPNeT) for visual-enhanced entity and relation extraction, aiming to achieve more effective and robust performance. Specifically, we regard visual representation as pluggable visual prefix to guide the textual representation for error insensitive forecasting decision. We further propose a dynamic gated aggregation strategy to achieve hierarchical multi-scaled visual features as visual prefix for fusion. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our method, and achieve state-of-the-art performance. Code is available in https://github.com/zjunlp/HVPNeT.
BibTeX
@misc{chen2022good,
  author = {Xiang Chen and Ningyu Zhang and Lei Li and Yunzhi Yao and Shumin Deng and Chuanqi Tan and Fei Huang and Luo Si and Huajun Chen},
  title = {{Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction}},
  year = {2022},
  eprint = {2205.03521},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.03521},
}
Figure