TL;DR
This paper introduces APKGC, a novel method for multimodal knowledge graph completion (MMKGC) that tackles the challenges of noise and "over-trust" in attention mechanisms. By adaptively sampling noisy multimodal data and penalizing overly confident attention scores, APKGC significantly improves the accuracy of predicting missing relationships in knowledge graphs.
What Problem Does This Paper Address?
Knowledge graphs (KGs) are powerful tools for organizing information, but they are often incomplete. Multimodal knowledge graphs (MMKGs) extend this by incorporating information from various modalities like text and images, offering a richer representation of the world. However, completing these MMKGs (predicting missing triples) faces two main hurdles:
- Lack of Robustness: Real-world data is inherently noisy. Existing MMKGC methods often struggle to handle this noise, leading to reduced model robustness.
- Attention Over-Trust: When using pre-trained language models (like BERT) for MMKGC, the attention mechanism can sometimes fixate on irrelevant or misleading parts of the input (e.g., punctuation), a phenomenon known as "attention partial over-trust." This can lead to inaccurate entity feature extraction and, consequently, poor prediction performance.
Methods
APKGC proposes two core innovations to address these problems:
- Adaptive Noise Sampling: Instead of trying to eliminate noise, APKGC embraces it. It adaptively samples noisy data from both text and image modalities during training. This adaptive strategy dynamically adjusts the amount and proportion of noisy data, simulating real-world conditions and enhancing the model's robustness. Two sampling strategies are employed: Gaussian noise sampling for generalization and average noise sampling for smoother noise.
- Attention Over-Trust Penalty: To combat the "attention over-trust" issue, APKGC introduces an attention penalty module. This module identifies tokens with excessively high attention scores and penalizes them. It then redistributes this "penalized" attention to surrounding tokens, encouraging the model to focus on more semantically important parts of the input. This process helps to correct attention errors and improve the learning of entity features.
These two modules work in conjunction with a standard entity feature extraction pipeline that uses pre-trained models (BERT for text, Beit for images) and a triple scoring function (RotatE) for prediction.
Datasets
The paper evaluates APKGC on two publicly available multimodal knowledge graph benchmarks:
- DB15K: Derived from DBPedia and augmented with images.
- MKG-W: Comprises subsets of Wikidata and YAGO.
The paper provides statistics for these datasets, including the number of entities, relations, and splits for training, validation, and testing.
Key Insights and Ideas
- Noise as a Feature, Not a Bug: The paper challenges the conventional approach of solely trying to filter out noise. By adaptively incorporating noisy data, APKGC demonstrates that noise can actually improve model robustness.
- Targeted Attention Correction: The "attention over-trust" problem is a subtle but significant issue in multimodal language models. The proposed penalty mechanism offers a direct way to mitigate this by rebalancing attention scores, leading to more accurate feature representations.
- Synergy of Modalities: The results strongly suggest that leveraging all available modalities (text and visual) is crucial for optimal MMKGC performance, with significant drops in performance when either modality is excluded.
Key Results
APKGC achieves state-of-the-art (SOTA) performance on both the DB15K and MKG-W datasets. Specifically:
- DB15K: Achieves a 3.3% improvement in Hits@1 compared to the baseline.
- MKG-W: Achieves a 3.4% improvement in Hits@1 compared to the baseline.
Ablation studies confirm the effectiveness of both the adaptive noise sampling and the attention penalty modules, with the model performance degrading significantly when either is removed.
Limitations and Future Work
The authors acknowledge that their work focuses on text and visual modalities. They suggest that future research could explore incorporating additional modalities like video and audio data into MMKGC models. This would further enrich the representations and potentially lead to even more comprehensive knowledge graph completion.
Summary updated