Cross-modal Multi-task Learning for Multimedia Event Extraction

Jianwei Cao, Yanli Hu, Zhen Tan, Xiang ZhaoAAAI 2025 · DOI

TL;DR

This paper introduces X-MTL, a novel cross-modal multi-task learning framework designed to improve multimedia event extraction. By jointly learning four related tasks (trigger detection, argument extraction, verb classification, and role classification) and employing a pseudo-label-based knowledge distillation with dynamic weight adjustment, X-MTL effectively bridges the gap between different modalities and significantly outperforms existing state-of-the-art methods.

What Problem Does This Paper Address?

The proliferation of multimedia content, especially news articles with accompanying images and videos, presents a rich source of information. However, extracting structured event knowledge from these multimodal documents is challenging. A key hurdle is establishing correlations between different modalities (text and images) during training, especially since obtaining fully annotated multimodal event data is expensive and time-consuming. Existing methods often rely on unlabeled image-text pairs, but the learned correlations may lack specific event knowledge, limiting their effectiveness for event-centric tasks. This paper tackles this challenge by proposing a framework that leverages multi-task learning to learn these cross-modal correlations at the task level, thereby improving comprehension and utilization of events in multimedia content.

Methods

The core of X-MTL is a cross-modal multi-task learning framework that simultaneously addresses four key tasks: trigger detection, argument extraction, verb classification, and role classification. To handle inputs from different modalities and tasks, the framework employs a hard parameter sharing based cross-modal multi-task model. This model consists of two modality-specific encoders (BERT for text and a CLIP vision encoder for images) and a modality-shared encoder (a stack of transformer blocks) to learn joint task representations.

To enrich and unify task inputs, X-MTL introduces textual and visual prompt learning methods. These methods help to better represent arguments by incorporating them into the input with special markers, preserving both local and global semantic information.

A significant challenge in multi-task learning is the potential for task conflict, especially across modalities with differing characteristics (the "modality gap"). To mitigate this, X-MTL proposes a pseudo-label based knowledge distillation method. This involves training separate single-task models, using them to generate pseudo-labels on an image-text pair dataset, and then transferring this knowledge to the multi-task model. This process is further enhanced by a dynamic weight adjustment method that dynamically adjusts task weights based on task-specific losses, preventing any single task from dominating and improving overall training stability.

Datasets

The paper utilizes the following datasets:

  • ACE 2005: Primarily used for textual event extraction training.
  • imSitu: Used for visual event extraction training.
  • VOA Caption: An unlabeled image-text pair dataset used for learning cross-modal correlations and generating pseudo-labels.
  • M2E2 (Multimedia Event Extraction benchmark): Used for evaluating the performance of the proposed framework. The paper notes that imSitu was refined using the M2E2 schema for better evaluation of visual event extraction.

Key Insights and Ideas

  • Task-level cross-modal correlations are crucial: The paper argues that learning correlations at the task level, rather than just through general image-text alignment, is more effective for event-centric tasks.
  • Prompt-based argument representation is beneficial: Using textual and visual prompts to represent arguments helps preserve richer semantic information, leading to better performance, especially in the multi-task setting.
  • Pseudo-label knowledge distillation is key to overcoming task conflict: Transferring knowledge from well-trained single-task models via pseudo-labels helps the multi-task model maintain performance on individual tasks and learn more robust cross-modal representations.
  • Dynamic weight adjustment prevents task dominance: This technique is essential for balancing the learning process in multi-task settings, preventing catastrophic forgetting and improving overall performance.

Key Results

X-MTL demonstrates significant improvements on the M2E2 benchmark:

  • Multimedia Event Mention: Surpasses current state-of-the-art (SOTA) methods by 4.1%.
  • Multimedia Argument Role: Surpasses current SOTA methods by 8.2%.
  • The framework also shows strong performance in textual and visual event extraction individually, outperforming methods like CAMEL.
  • Ablation studies confirm the effectiveness of both the pseudo-label knowledge distillation and dynamic weight adjustment strategies.

Limitations and Future Work

The authors acknowledge that X-MTL is affected by error propagation from pseudo labels, which can impact performance. Additionally, the overall training process is described as relatively complex.

For future work, the authors suggest exploring more efficient optimization methods for cross-modal multi-task learning and broadening the application scope of their framework to other multimodal tasks.

Summary updated

BibTeX
@inproceedings{cao2025cross,
  author = {Jianwei Cao and Yanli Hu and Zhen Tan and Xiang Zhao},
  title = {{Cross-modal Multi-task Learning for Multimedia Event Extraction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i11.33246},
}
Figure