Introduction
Extracting structured information about real-world events—such as who did what to whom, where, and with what instrument—is a cornerstone of automated news analysis. Traditionally, Event Extraction (EE) has been treated as a unimodal task: Natural Language Processing (NLP) models analyze text, while Computer Vision (CV) models identify actions in video. However, news is inherently multimedia. A text article often provides high-level context, while the accompanying video captures specific, dynamic details that the text might omit.
Researchers have identified a significant information gap: approximately 45% of news videos contain event arguments (participants or objects) that are not explicitly mentioned in the associated text. To bridge this gap, this work introduces the task of Video MultiMedia Event Extraction (Video M2E2) and proposes a novel framework to jointly extract events from both video and text articles.
The Video M2E2 Task and Dataset
The goal of Video M2E2 is to build a comprehensive understanding of an event by "reading" an article and "watching" its corresponding video simultaneously. This requires the system to not only identify events in each modality but also to ground textual mentions in visual evidence.
To facilitate research in this area, the authors developed the Video M2E2 dataset. This benchmark consists of 860 video-article pairs sourced from major news outlets like the BBC, VOA, and Reuters. It covers 16 diverse event types (e.g., Conflict.Attack, Justice.ArrestJail, Disaster.FireExplosion) and includes extensive annotations: 852 coreference links between text and video, and over 24,000 annotated argument roles grounded with bounding boxes in the video frames.
Methodology: Joint Multimodal Extraction
The proposed framework consists of two primary components designed to handle the complexities of cross-modal alignment and extraction.
1. Self-Supervised Multimodal Event Coreference
Before detailed extraction can occur, the system must determine which sentences in an article correspond to which segments of the video. Because manual labels for such pairs are scarce, the authors developed a self-supervised coreference model. This model uses auto-generated speech transcripts (ASR) as a bridge to learn a shared embedding space for text and video. A key innovation is the "Multi-Instance Learning from Objects" (MILO) loss, which encourages the model to align text mentions of specific objects (e.g., "van") with video segments where those objects are visually detected.
2. Joint Multimodal Transformer (JMMT)
Once coreferential segments are identified, they are processed by the Joint Multimodal Transformer (JMMT). This encoder-decoder architecture is designed to fuse information across modalities.
- Encoder: The encoder fuses textual tokens with a rich set of visual features, including global video context, object labels, region-specific features, and spatial coordinates (bounding boxes).
- Decoder: The decoder employs two specialized heads. One head identifies "triggers" and entities within the text, while the other classifies video event types and predicts bounding boxes for visual arguments.
Key Insights: Modality Complementarity
The research highlights several ways in which joint processing improves extraction accuracy:
- Temporal Context as a Disambiguator: Static images can be ambiguous. For instance, in a photo of an arrest, it may be difficult to distinguish the "Jailer" from the "Detainee." Video dynamics—such as the specific motion of applying handcuffs—provide the temporal context necessary to assign these roles correctly.
- Filling the "Missing Argument" Gap: By leveraging both modalities, the model can recover information missing from one. If a text describes "Officers driving into protesters" without specifying the vehicle, the JMMT can extract "Vans" as the Instrument role based solely on the visual data.
- Object-Centric Alignment: The authors found that aligning specific objects to corresponding words is more effective than trying to align entire sentences to entire video clips. This granular alignment significantly boosts the accuracy of the coreference stage.
Experimental Results
The JMMT model demonstrates significant improvements over unimodal and state-of-the-art image-text baselines:
- Coreference Resolution: The model achieved a 6.0% absolute F-score gain in matching sentences to video segments.
- Event Extraction: The system saw a 5.8% absolute F-score gain in overall multimedia event extraction.
- Relative Gains: In specific event categories, the joint model produced up to a 74% relative improvement over text-only or video-only baselines, confirming that the modalities provide complementary, non-redundant information.
Conclusion and Future Directions
This work represents the first successful attempt to jointly extract structured event information from video and text articles. While highly effective, the authors note that the current model assumes a close relationship between the video and the article, which may not hold for loosely related "in-the-wild" web content. Future work will explore incorporating entity tracking to maintain participant identities across long video sequences, potentially improving role labeling when subjects are temporarily obscured or move out of frame.
Summary updated