Abstract
Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji, Shih-Fu Chang. Findings of the Association for Computational Linguistics: EMNLP 2021.
2021.
Introduction
In modern digital journalism, news stories are rarely delivered through a single medium. A written article typically provides high-level context and background, while an accompanying video captures the dynamic, visual details of the event. Despite this synergy, traditional Event Extraction (EE) systems—designed to identify "who did what to whom"—have largely operated in silos. Text-based models ignore visual evidence, and computer vision models often struggle with the abstract concepts and entities defined in text.
Research indicates a significant information gap: approximately 45% of news videos contain event arguments (participants or locations) that are never explicitly mentioned in the accompanying text. To bridge this gap, the authors introduce the task of Video MultiMedia Event Extraction ($VM^2E^2$). The goal is to extract a unified "event frame" that integrates information from both modalities, ensuring a more complete understanding of complex news events.
The $VM^2E^2$ Framework
The proposed system addresses the challenges of cross-modal information fusion through a two-stage pipeline that first aligns the modalities and then extracts structured information jointly.
Stage 1: Multimodal Event Coreference Resolution
Before extracting specific details, the system must align the article's narrative with the video's timeline. This stage determines which sentence in the text corresponds to which segment of the video. The authors employ a self-supervised approach using Noise Contrastive Estimation (NCE) loss, which does not require manually labeled pairs for training.
A key innovation here is the Multi-Instance Learning from Objects (LMILO) loss. Recognizing that global video features can be noisy, LMILO encourages the model to focus on specific objects. For instance, if a sentence mentions an "Attack," the model learns to look for visual evidence like "vans" or "protesters" to confirm the coreference link between the text and the video segment.
Stage 2: Joint Multimodal Transformer (JMMT)
Once alignment is established, the Joint Multimodal Transformer (JMMT) processes the paired data.
- Encoder: Fuses text tokens with diverse visual features, including video-level context, object labels, region-level features from bounding boxes, and spatial coordinates.
- Decoder: Employs dual heads to simultaneously extract triggers and entities from the text while classifying video event types and predicting bounding boxes for visual arguments.
Dataset and Evaluation
To support this new task, the authors curated the $VM^2E^2$ dataset, comprising 860 video-article pairs from major news outlets like VOA, BBC, and Reuters. The dataset covers 16 visually detectable and frequent event types, such as Conflict.Attack and Movement.Transport. It includes detailed annotations for temporal boundaries, coreferential links, and bounding boxes for participants.
Key Findings and Results
The JMMT model significantly outperformed unimodal baselines across all metrics:
- Coreference Accuracy: Achieved a 51.5 F1 score, surpassing standard self-supervised models like HowTo100M.
- Extraction Performance: The joint model reached a 47.6 F1 score for event mentions and 21.3 F1 for argument roles in the multimedia setting.
- Synergistic Gains: Compared to text-only or video-only models, JMMT showed up to a 74% relative gain in extraction accuracy.
These results highlight the power of cross-modal discovery. The system can identify arguments present in only one modality—such as a specific street name visible in the video but omitted from the text—leading to a richer, more accurate event representation.
Conclusion and Future Work
This work establishes a foundation for joint multimedia event extraction by proving that "seeing" the video helps the model understand the text better, and vice versa. While the results are promising, challenges remain in handling highly chaotic footage and maintaining participant "memory" during visual occlusions. Future research will focus on incorporating temporal entity tracking to improve robustness in unscripted or complex video environments.
Summary updated