Multimedia Event Extraction with LLM Knowledge Editing

Jiaao Yu, Yijing Lin, Zhipeng Gao, Xuesong Qiu, Lanlan RuiEMNLP 2025

TL;DR

This paper tackles the challenge of multimedia event extraction (MEE) by proposing a novel knowledge editing approach for large language models (LLMs). Instead of traditional fine-tuning, they identify and edit "redundant" visual features within LLMs that hinder accurate event understanding, leading to state-of-the-art performance on the M2E2 benchmark without requiring additional training data or parameters.

What Problem Does This Paper Address?

Multimedia Event Extraction (MEE) aims to identify event types and their associated arguments (like participants, locations, etc.) from both text and images. While LLMs have shown great promise, applying them to MEE is challenging due to the high cost of creating labeled multimedia datasets. Existing methods often rely on weakly aligning features from pre-trained unimodal encoders (text and image models). However, these methods can suffer from a fundamental conflict: image recognition models are trained to capture as many visual features as possible, which can lead to "over-understanding" or focusing on irrelevant details. This can confuse the LLM when it needs to extract specific event information, especially when visual cues are ambiguous or misleading, as illustrated by the example of a person running away being misidentified as an arrest. This paper addresses this issue by proposing a method to refine the LLM's visual understanding for MEE tasks.

Methods

The core of this paper's contribution lies in its novel knowledge editing strategy applied to the visual perception layers of multimodal LLMs (MLLMs). The approach involves two main stages:

  1. Multilevel Redundant Neuron Selection: This mechanism aims to identify visual neurons that capture redundant or irrelevant features for event extraction. It first converts network weights into a probability distribution and then uses entropy to measure the importance of neural network layers. By analyzing the differences in weight distributions between neighboring layers, they identify layers that are overly sensitive to feature learning. Within these identified layers, they use the L1-norm to pinpoint specific neurons that are likely capturing redundant information. A threshold (δ) is used to avoid collapsing the layer.
  2. Mask Matrix-Based Knowledge Editing: Once redundant neurons are identified, a mask matrix is constructed. This matrix is used to selectively zero out the weights connected to these redundant neurons in the visual layers of the MLLM. This effectively "edits" the LLM's knowledge by removing or downplaying the influence of irrelevant visual features, forcing it to focus on more pertinent information for event understanding. Crucially, this process does not require backpropagation or additional parameter optimization, making it computationally efficient.

Datasets

The primary dataset used for evaluation is M2E2 (Multimedia Event Extraction dataset). This benchmark contains 245 multimedia documents, 6167 sentences, and 1014 images, covering 8 event types and 15 argument types. The paper notes that M2E2 primarily provides data for testing, which limits fine-tuning possibilities.

Key Insights and Ideas

  • Visual Over-Understanding is a Bottleneck: The paper highlights that the very nature of training visual encoders for general image recognition can be detrimental to specific tasks like MEE, where precise feature selection is key.
  • Knowledge Editing for Feature Refinement: Applying knowledge editing not to factual knowledge but to the perception capabilities of an LLM is a novel direction. By editing visual features, they indirectly improve textual argument extraction as well.
  • Efficiency Through No Fine-Tuning: The proposed method achieves SOTA performance without requiring any fine-tuning or additional trainable parameters, making it highly efficient in terms of computation and storage.
  • Entropy and L1-Norm for Redundancy Detection: The combination of entropy for layer importance and L1-norm for neuron identification provides a principled way to find problematic visual features.

Key Results

The proposed method achieves state-of-the-art (SOTA) performance on the M2E2 benchmark.

  • Event Extraction: The paper reports a 34% improvement in Precision for multimedia event extraction compared to the highest baseline.
  • Argument Extraction: A 11% improvement in F1 score is achieved for multimedia argument extraction.
  • Compared to the original Qwen2-VL-7b model, the proposed method shows significant gains, particularly in multimedia event extraction, with improvements of 6.15% Precision, 14.56% Recall, and 16.75% F1.
  • Experiments with variants (Variant-1, -2, -3) demonstrate the effectiveness of their specific approach to redundant neuron selection and editing.

Limitations and Future Work

The authors acknowledge several limitations:

  • Image-Only Argument Extraction: While the method improves overall MEE, its sensitivity to image-only argument extraction is less pronounced. They hypothesize this might be due to MLLMs' tendency towards object-level bounding boxes rather than instance-level precision.
  • Dataset Limitations: The M2E2 dataset itself has some missing argument labels.
  • Event Type Granularity: The paper notes that events can be further subdivided (e.g., "Justice:Arrest-Jail" vs. "Contact:Meet"), and these distinctions might require different mining difficulties, which is not fully addressed.

Future work suggested includes exploring domain adaptation capabilities for multimodal LLMs.

Summary updated

BibTeX
@inproceedings{yu2025multimedia,
  author = {Jiaao Yu and Yijing Lin and Zhipeng Gao and Xuesong Qiu and Lanlan Rui},
  title = {{Multimedia Event Extraction with LLM Knowledge Editing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}

Loading related content...

Figure