A Hierarchical Network for Multimodal Document-Level Relation Extraction

Lingxing Kong, Jiuliang Wang, Zheng Ma, Qifeng Zhou, Jianbing Zhang, Liang He, Jiajun ChenAAAI 2024 · DOI

Introduction

Information extraction has traditionally focused on text-only content. However, real-world documents, particularly in news and social media, are increasingly multimodal. This paper, presented at AAAI-24, introduces Multimodal Document-level Relation Extraction (MDocRE). This new task leverages both textual and video data to identify complex relationships between entities, moving beyond the limitations of unimodal systems.

The MDocRE Task and Challenges

Document-level Relation Extraction (DocRE) is inherently more difficult than sentence-level extraction due to two primary factors:

  1. Long-range Dependencies: Relevant entities may be mentioned across distant paragraphs, requiring models to maintain long-term context.
  2. Mention Selection: Entities often have multiple mentions throughout a document, but only a subset of these mentions typically provides the evidence required to establish a specific relationship.
Overview of the MDocRE task, illustrating how visual cues from video resolve textual ambiguities to identify the 'Competitor' relation.

The authors propose that video serves as a "wider perspective." Visual interactions, facial expressions, and shared scenes can bridge textual gaps and help filter out irrelevant mentions. For example, while text might use rhetorical devices like irony that obscure a relationship, a video of two people in a heated debate provides unambiguous evidence of a "tense relationship."

Methodology: Hierarchical Fusion

The core of the proposed approach is a hierarchical network designed to fuse modalities at multiple levels of granularity:

  • Hierarchical Encoding: The model employs a Local Encoder to align intra-frame visual structures with textual phrases and a Global Encoder to align inter-frame visual bindings with high-level document semantics.
  • Textual-guided Transformer: Unlike symmetric fusion methods, this architecture uses textual features as "prompts" to guide the learning of visual embeddings. This design choice reflects the fact that in news documents, text remains the primary carrier of logical structure.
  • Mention Gate Module: This novel attention mechanism calculates the importance of each entity mention by analyzing both intra-modality and inter-modality correlations. This effectively "gates" out noisy or irrelevant mentions that do not contribute to relation identification.

Experimental Results

To evaluate their approach, the researchers introduced the MDocRE Dataset, consisting of 3,043 human-annotated documents and corresponding VOA news videos. The dataset is characterized by long videos (average 210 seconds) and 21 distinct relation types.

The proposed model achieved state-of-the-art performance, reaching an F1 score of 40.54. This significantly outperformed the best unimodal baseline (ATLOP at 36.89) and specialized multimodal models like ClipBERT (20.88). Ablation studies highlighted the critical role of the Mention Gate, which provided a 2.02 F1 point boost. Furthermore, the researchers demonstrated that even top-tier unimodal models saw performance gains when integrated into this multimodal hierarchical framework.

Conclusion and Future Work

By treating video as a "second opinion" and utilizing an asymmetric, text-guided fusion strategy, the proposed hierarchical network successfully addresses the ambiguities inherent in long-form text. While the model currently struggles with "overlapping relations"—where entity pairs share multiple simultaneous relationships—future work will focus on multi-label detection and expanding the dataset to more diverse domains such as educational content and film.

Summary updated

BibTeX
@inproceedings{kong2024hierarchical,
  author = {Lingxing Kong and Jiuliang Wang and Zheng Ma and Qifeng Zhou and Jianbing Zhang and Liang He and Jiajun Chen},
  title = {{A Hierarchical Network for Multimodal Document-Level Relation Extraction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29801},
}
Figure