Multi-source Knowledge Enhanced Graph Attention Networks for Multimodal Fact Verification

Han Cao, Lingwei Wei, Wei Zhou, Songlin Hu

TL;DR

This paper introduces MultiKE-GAT, a novel graph-based approach for multimodal fact verification that effectively fuses information from diverse sources like text and images. By constructing a heterogeneous graph and employing a knowledge-aware fusion module, MultiKE-GAT significantly improves accuracy in determining the truthfulness of claims, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Multimodal fact verification, the task of assessing the veracity of claims using evidence from multiple modalities (like text and images), is a growing area of research. The core challenge lies in effectively integrating these diverse pieces of information to form a comprehensive understanding. Existing methods often struggle with fusing features from different modalities, leading to a loss of fine-grained details and an inability to handle noise or inconsistencies introduced by irrelevant information. This paper aims to bridge this gap by proposing a method that can leverage multi-source, fine-grained knowledge to achieve more accurate and robust fact verification.

Methods

The proposed model, MultiKE-GAT (Multi-source Knowledge-enhanced Graph Attention Network), tackles the problem through several key innovations:

  • Multi-source Knowledge Extraction: The model first extracts various types of knowledge:
  • Textual Entities: Using tools like TAGME to identify key entities in claim and evidence text.
  • Visual Entities: Employing Faster R-CNN to detect objects in images.
  • Key Information: Leveraging Large Language Models (LLMs) like GPT-3.5 to extract crucial phrases and summaries from text. Thresholds are used to filter out noisy or irrelevant extracted information.

  • Heterogeneous Graph Construction: The extracted knowledge is organized into a heterogeneous graph. This graph comprises:

  • Global Nodes: Representing coarse-grained textual and visual features of the claim and evidence.
  • Textual Entity Nodes: Representing extracted textual entities.
  • Visual Entity Nodes: Representing detected visual objects.
  • Key Information Nodes: Representing extracted key phrases from LLMs. Edges connect these nodes, with weights determined by cosine similarity, capturing both intra- and inter-modal relationships.

  • Knowledge-oriented Graph-based Fusion (KGF): This module is central to MultiKE-GAT's success. It uses graph attention mechanisms to iteratively update node embeddings, allowing for the capture of complex interactions between different types of nodes. A key aspect is the use of "global nodes" to guide the fusion process, helping to emphasize relevant entities and suppress irrelevant ones, thereby mitigating noise and inconsistencies.

  • Classifier: Finally, aggregated node features from the graph are fed into a simple MLP classifier to predict the claim's veracity (e.g., supported, refuted).

Datasets

The paper evaluates MultiKE-GAT on two public multimodal fact verification benchmarks:

  • FACTIFY: This dataset contains 42,500 multimodal claim-evidence pairs across five categories: Support Multimodal, Support Text, Insufficient Multimodal, Insufficient Text, and Refute. The authors randomly split the original training set into 80% for training and 20% for validation/testing.
  • MOCHEG: This dataset comprises 21,184 claims and evidence, categorized into Supported, Refuted, and Not Enough Info (NEI). The original partition of this dataset is used for training, validation, and testing.

Key Insights and Ideas

  • The Power of Fine-Grained, Multi-Source Knowledge: The paper strongly emphasizes that simply fusing coarse-grained multimodal features is insufficient. Integrating fine-grained entities and key information from diverse sources (text, images, LLMs) is crucial for accurate fact verification.
  • Heterogeneous Graphs for Complex Interactions: Representing multimodal data as a heterogeneous graph allows for a more nuanced capture of relationships between different types of information (textual entities, visual objects, global features).
  • Global Guidance for Noise Reduction: The "Knowledge-aware Graph Fusion" module's use of global nodes as a guiding mechanism is a clever way to filter out irrelevant information and focus on the most pertinent clues, a significant challenge in multimodal tasks.
  • LLMs as Knowledge Augmenters: The integration of LLMs to extract key information highlights their growing role in providing valuable contextual data for downstream tasks like fact verification.

Key Results

MultiKE-GAT demonstrates state-of-the-art performance on both the FACTIFY and MOCHEG datasets.

  • FACTIFY: MultiKE-GAT achieves a weighted-average F1 score of 79.64% on the 5-way task and 83.68% on the 3-way task, significantly outperforming various unimodal and multimodal baselines. For instance, it surpasses the next best multimodal method (Logically+) by over 2.6% on the 5-way task.
  • MOCHEG: The model achieves an F1 score of 70.14%, also outperforming existing methods.
  • Ablation Studies: These studies confirm the importance of each component: removing multi-source knowledge or the graph fusion module leads to significant performance drops, underscoring the effectiveness of the proposed approach.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their model effectively filters noise, the introduction of entities can still bring in some noise that might hinder performance. The reliance on external toolkits for knowledge extraction also means the quality of these tools can impact the overall results.
  • Future Work: The authors plan to explore how external knowledge can be used to provide better explanations for multimodal fact verification. This suggests a direction towards more interpretable AI systems in this domain.

Summary updated

Abstract

Multimodal fact verification is an under-explored and emerging field that has gained increasing attention in recent years. The goal is to assess the veracity of claims that involve multiple modalities by analyzing the retrieved evidence. The main challenge in this area is to effectively fuse features from different modalities to learn meaningful multimodal representations. To this end, we propose a novel model named Multi-Source Knowledge-enhanced Graph Attention Network (MultiKE-GAT). MultiKE-GAT introduces external multimodal knowledge from different sources and constructs a heterogeneous graph to capture complex cross-modal and cross-source interactions. We exploit a Knowledge-aware Graph Fusion (KGF) module to learn knowledge-enhanced representations for each claim and evidence and eliminate inconsistencies and noises introduced by redundant entities. Experiments on two public benchmark datasets demonstrate that our model outperforms other comparison methods, showing the effectiveness and superiority of the proposed model.
BibTeX
@inproceedings{cao2024multi,
  author = {Han Cao and Lingwei Wei and Wei Zhou and Songlin Hu},
  title = {{Multi-source Knowledge Enhanced Graph Attention Networks for Multimodal Fact Verification}},
  booktitle = {ICME},
  year = {2024},
  url = {https://openreview.net/forum?id=2XZkWybW6x},
}
Figure