Introduction
Information extraction has traditionally focused on text-only content. However, real-world documents, particularly in news and social media, are increasingly multimodal. This paper, presented at AAAI-24, introduces Multimodal Document-level Relation Extraction (MDocRE). This new task leverages both textual and video data to identify complex relationships between entities, moving beyond the limitations of unimodal systems.
The MDocRE Task and Challenges
Document-level Relation Extraction (DocRE) is inherently more difficult than sentence-level extraction due to two primary factors:
- Long-range Dependencies: Relevant entities may be mentioned across distant paragraphs, requiring models to maintain long-term context.
- Mention Selection: Entities often have multiple mentions throughout a document, but only a subset of these mentions typically provides the evidence required to establish a specific relationship.
The authors propose that video serves as a "wider perspective." Visual interactions, facial expressions, and shared scenes can bridge textual gaps and help filter out irrelevant mentions. For example, while text might use rhetorical devices like irony that obscure a relationship, a video of two people in a heated debate provides unambiguous evidence of a "tense relationship."
Methodology: Hierarchical Fusion
The core of the proposed approach is a hierarchical network designed to fuse modalities at multiple levels of granularity:
- Hierarchical Encoding: The model employs a Local Encoder to align intra-frame visual structures with textual phrases and a Global Encoder to align inter-frame visual bindings with high-level document semantics.
- Textual-guided Transformer: Unlike symmetric fusion methods, this architecture uses textual features as "prompts" to guide the learning of visual embeddings. This design choice reflects the fact that in news documents, text remains the primary carrier of logical structure.
- Mention Gate Module: This novel attention mechanism calculates the importance of each entity mention by analyzing both intra-modality and inter-modality correlations. This effectively "gates" out noisy or irrelevant mentions that do not contribute to relation identification.
Experimental Results
To evaluate their approach, the researchers introduced the MDocRE Dataset, consisting of 3,043 human-annotated documents and corresponding VOA news videos. The dataset is characterized by long videos (average 210 seconds) and 21 distinct relation types.
The proposed model achieved state-of-the-art performance, reaching an F1 score of 40.54. This significantly outperformed the best unimodal baseline (ATLOP at 36.89) and specialized multimodal models like ClipBERT (20.88). Ablation studies highlighted the critical role of the Mention Gate, which provided a 2.02 F1 point boost. Furthermore, the researchers demonstrated that even top-tier unimodal models saw performance gains when integrated into this multimodal hierarchical framework.
Conclusion and Future Work
By treating video as a "second opinion" and utilizing an asymmetric, text-guided fusion strategy, the proposed hierarchical network successfully addresses the ambiguities inherent in long-form text. While the model currently struggles with "overlapping relations"—where entity pairs share multiple simultaneous relationships—future work will focus on multi-label detection and expanding the dataset to more diverse domains such as educational content and film.
Summary updated