Abstract
Multimodal Information Extraction (MIE) aims to enhance text-based entity and relation extraction by leveraging associated visual information. However, existing methods often struggle with "noise"—irrelevant images that degrade performance compared to text-only baselines. This paper introduces HVPNeT (Hierarchical Visual Prefix fusion NeTwork), a novel architecture that treats multi-scale visual features as hierarchical prefixes to guide the attention mechanism of a pre-trained language model. By utilizing a dynamic gated aggregation mechanism, HVPNeT effectively filters irrelevant visual noise while capturing multi-level cross-modal alignments. Experimental results demonstrate that HVPNeT achieves state-of-the-art performance on MNER and MRE benchmarks and exhibits superior robustness and data efficiency.
Introduction
In the domain of Information Extraction (IE), the integration of multimodal data—specifically text and images—has become a vital frontier. Tasks such as Multimodal Named Entity Recognition (MNER) and Multimodal Relation Extraction (MRE) rely on the premise that visual context can disambiguate textual references. For instance, an image of a basketball court can clarify that a mention of "LeBron" refers to the professional athlete rather than a different individual.
Despite the potential of multimodal models, current architectures like UMGF or MEGA face a significant hurdle: error sensitivity. In social media contexts, images are frequently only tangentially related or entirely irrelevant to the accompanying text. When faced with such noise, many existing models perform worse than their text-only counterparts. The challenge lies in developing a system that can leverage helpful visual clues without being distracted by irrelevant ones.
Methodology
HVPNeT addresses these challenges by moving away from traditional feature concatenation, instead adopting a "prefix-tuning" inspired approach to multimodal fusion.
1. Visual Prefix-guided Fusion
Rather than simply appending visual vectors to textual embeddings, HVPNeT treats visual features as "pluggable" prefixes. These prefixes are prepended to the keys ($K$) and values ($V$) of every self-attention layer within a pre-trained BERT model. This allows the visual data to function as a prompt, modulating how the textual representation is formed at each layer of the transformer.
2. Hierarchical Pyramidal Features
Linguistic abstraction in models like BERT varies across layers, moving from surface-level syntax to deep semantics. To match this, HVPNeT extracts visual features from multiple blocks of a ResNet backbone, creating a "pyramid" of visual information. This ensures that fine-grained textures and global object shapes are available to the textual encoder at the appropriate stages of processing.
3. Dynamic Gated Aggregation
To manage the interaction between different visual scales and textual layers, the model employs a learnable gating mechanism. This mechanism adaptively weights and aggregates multi-scale features, effectively "routing" the most relevant visual information to the corresponding level of text processing. This gating is crucial for robustness, as it allows the model to suppress irrelevant visual features that do not align with the text.
Experimental Evaluation
The performance of HVPNeT was evaluated on three standard multimodal benchmarks:
- Twitter-2015 & Twitter-2017: Benchmarks for Multimodal Named Entity Recognition (MNER).
- MNRE: The primary benchmark for Multimodal Relation Extraction (MRE), consisting of over 9,000 image-text pairs.
The implementation utilized ResNet50 as the visual backbone and BERT-base as the textual encoder.
Key Results
HVPNeT established new state-of-the-art (SOTA) benchmarks across all tested datasets:
- MNER: On the Twitter-2017 dataset, the model achieved an F1 score of 86.87%, representing a 1.36% improvement over the previous best model (UMGF).
- MRE: On the MNRE dataset, HVPNeT reached an F1 score of 81.85%, a substantial improvement over the previous SOTA (MEGA at 66.41%).
- Robustness: Cross-task generalization tests showed that HVPNeT experienced significantly less performance degradation than previous models when moved across different datasets.
- Efficiency: In low-resource scenarios (using only 5% to 10% of training data), HVPNeT maintained a significant performance lead, suggesting high data efficiency.
Discussion and Insights
The success of HVPNeT stems from two core insights. First, treating images as prefixes rather than input tokens allows the model to treat visual data as context that modulates the text. Second, the alignment of scales—matching low-level visual features with syntactic layers and high-level concepts with semantic layers—provides a more nuanced fusion than flat concatenation. The dynamic gating mechanism serves as a critical filter, solving the long-standing problem of multimodal models being less reliable than text-only models in the presence of noise.
Limitations and Future Work
While HVPNeT represents a significant advancement, it currently focuses on unidirectional guidance (vision guiding text). Future research could explore bidirectional guidance to improve computer vision tasks using textual prompts. Additionally, the model does not currently perform visual grounding (linking entities to specific pixels). Extending this hierarchical prefix approach to large-scale multimodal pre-training remains a promising avenue for future development.
Summary updated