Coreference by Appearance: Visually Grounded Event Coreference Resolution

Liming Wang, Shengyu Feng, Xudong Lin, Manling Li, Heng Ji, Shih‐Fu Chang2021 · DOI

Abstract

Event coreference resolution is critical to understand events in growing number of online news with multiple modalities including text, video, speech, etc. However, the events and entities depicting in different modalities may not be perfectly aligned and can be difficult to annotate, which makes the task especially challenging with little supervision available. To address the above issues, we propose a supervised model based on attention mechanism and an unsupervised model based on statistical machine translation, capable of learning the relative importance of modalities for event coreference resolution. Experiments on a video multimedia event dataset show that our multimodal models outperform text-only systems in the event coreference resolution task. A careful analysis reveals that the performance gain of the multimodal model especially under the unsupervised setting comes from better learning of visually salient events.

Introduction

In natural language processing, Event Coreference Resolution (ECR) is the task of determining if two mentions (e.g., "the blast" and "the explosion") refer to the same real-world occurrence. While humans naturally use multimodal context—such as seeing a video of a building collapsing to resolve an ambiguous "attack" mention—current AI systems are often restricted to text. These text-only models are typically "data-hungry" and struggle with linguistic ambiguity. This research explores giving AI a visual "reality check" by grounding text mentions in accompanying video footage.

Methodology

The researchers propose two distinct approaches to integrate video into the ECR pipeline:

Supervised Weight Fusion Model

This model utilizes RoBERTa for text embeddings and a Temporal Segment Network (TSN) for video features. A cross-media attention mechanism allows the model to "attend" to relevant video segments for each text mention, determining which visual cues best match the linguistic description.

Unsupervised Amodal SMT Model

To address the high cost of multimodal annotation, the authors developed an unsupervised model based on Statistical Machine Translation (SMT) principles. It treats the relationship between text and video as a translation problem, operating in three distinct modes:

  • Visual Mode ($\pi_1$): Resolves mentions based on similarity to video segments.
  • Trigger-Matching Mode ($\pi_3$): Focuses on semantic word meanings.
  • Attribute-Matching Mode: Focuses on syntactic features like sentence distance.
The Amodal SMT model architecture, showing how video segments (Z) and text mentions (M) are linked through different modes like visual and trigger matching.

The model uses an Expectation-Maximization (EM) algorithm to learn the optimal mode for a given mention without human-labeled coreference links.

Experimental Setup

The models were evaluated using the Video M2E2 dataset, which contains 860 news documents paired with 1–2 minute videos from BBC and Voice of America. The dataset includes over 4,000 event mentions across 647 clusters, representing a challenging real-world environment where text and video are not perfectly aligned.

Results and Discussion

The multimodal approach consistently outperformed text-only baselines across both supervised and unsupervised settings.

Performance Gains

  • Supervised: The "Weight Fusion" model achieved a 61.6 Pairwise F1 score, surpassing the text-only state-of-the-art (61.0).
  • Unsupervised: The "Amodal SMT" model reached 57.7 Pairwise F1, a significant improvement over the text-only SMT baseline (54.1).

Visual Grounding as a Tie-Breaker

Video context is most effective when text is linguistically diverse. For instance, a text-only model might fail to link "blew out" with "explosion" due to dissimilar word embeddings. Visual grounding provides the "connective tissue" to resolve these mentions.

Qualitative comparison between Text-only SMT and Amodal SMT. The amodal model successfully links linguistically diverse mentions like 'blew out' to 'explosion' using visual context.

Visual Salience

The performance boost is most pronounced for "visually salient" events—actions that are clearly visible, such as Demonstrate or TransportPerson. For the Demonstrate event type, the amodal SMT model improved performance by 18% over the text-only baseline.

Conclusion and Future Work

While multimodal grounding significantly improves ECR for visually distinct events, it faces challenges with "non-visual" or rare events where visual evidence may be metaphorical or absent. Future work aims to integrate global clustering mechanisms to improve document-level consistency and prevent locally logical but globally impossible coreference chains.

Summary updated

Abstract

Event coreference resolution is critical to understand events in growing number of online news with multiple modalities including text, video, speech, etc. However, the events and entities depicting in different modalities may not be perfectly aligned and can be difficult to annotate, which makes the task especially challenging with little supervision available. To address the above issues, we propose a supervised model based on attention mechanism and an unsupervised model based on statistical machine translation, capable of learning the relative importance of modalities for event coreference resolution. Experiments on a video multimedia event dataset show that our multimodal models outperform text-only systems in the event coreference resolution task. A careful analysis reveals that the performance gain of the multimodal model especially under the unsupervised setting comes from better learning of visually salient events.
BibTeX
@misc{wang2021coreference,
  author = {Liming Wang and Shengyu Feng and Xudong Lin and Manling Li and Heng Ji and Shih‐Fu Chang},
  title = {{Coreference by Appearance: Visually Grounded Event Coreference Resolution}},
  year = {2021},
  doi = {10.18653/v1/2021.crac-1.14},
}
Figure