Knowledge Graphs (KGs) serve as the foundation for modern search and recommendation systems, yet they are frequently incomplete. Multimodal Knowledge Graphs (MMKGs) attempt to address these gaps by augmenting traditional symbolic triples—head, relation, and tail—with auxiliary data such as images and text. However, MMKGs face a significant hurdle in Few-Shot Relation Learning (FSRL): the ability to learn a brand-new relation from only a handful of examples.
In the paper "FusionAdapter for Few-Shot Relation Learning in Multimodal Knowledge Graphs," researchers introduce a novel framework designed to integrate diverse data streams without sacrificing the unique characteristics of each modality.
The Multimodal Challenge: Over-Homogenization
Traditional KGs represent facts as symbolic triples, such as (Wichita Falls, Capital_of, Wichita County). MMKGs enrich these facts with visual and textual context.
Existing methods often struggle with two primary issues:
- Over-Homogenization: Many models attempt to "align" modalities by forcing image, text, and graph embeddings into the same mathematical space. This often results in the loss of modality-specific details that are crucial for distinguishing complex relations.
- The Few-Shot Bottleneck: Updating massive multimodal models for every new relation is computationally prohibitive and prone to overfitting when data is scarce.
Proposed Methodology: FusionAdapter
FusionAdapter addresses these challenges by building upon the MetaR framework and introducing two key innovations: the Modality Adapter and Contrastive Diversity Loss.
1. The Modality Adapter
Instead of retraining the entire entity embedding, FusionAdapter utilizes a lightweight Adapter Module. This module consists of a feed-forward network with a bottleneck structure inserted into the visual and textual processing streams. During meta-testing—when the model learns a new, unseen relation—only these small adapters are updated. This approach is highly parameter-efficient, adding only 0.22% to 0.75% more parameters to the base model.
2. Contrastive Diversity Loss
Perhaps the most significant conceptual shift in this work is the move from alignment to diversity. The authors introduce a diversity loss that penalizes the model if different modalities become too similar. By maintaining a "margin" of difference, the model is forced to extract complementary information—such as relational context from text and visual attributes from images—rather than redundant signals.
The final entity representation is calculated as a summation of the structural, adapted textual, and adapted visual embeddings: $$E = E_{Structure} + E'_{Text} + E'_{Visual}$$
Experimental Evaluation
The researchers evaluated FusionAdapter on three benchmark datasets: WN9-IMG (WordNet-based), FB-IMG (Freebase-based), and FB-IMG (F) (a filtered version to prevent relation leakage).
Key Results
- Performance Gains: FusionAdapter consistently outperformed state-of-the-art (SOTA) models. On the WN9-IMG dataset, it achieved a Mean Reciprocal Rank (MRR) of 0.327, a significant leap over MetaR’s 0.209.
- Modality Importance: Ablation studies revealed that text generally provides a larger performance boost than images, likely due to the richer relational semantics found in textual descriptions.
- Robustness: The model proved resilient to missing data. When an entity lacked an image or text, the adapters helped the model "fill in the blanks" more effectively than unimodal alternatives.
Conclusion
FusionAdapter demonstrates that rigid alignment between modalities can be counterproductive. By allowing different data streams to remain distinct, the model captures a more nuanced understanding of entities and their relations. For the future of knowledge graph completion, this research suggests that the most effective way to fuse data is to ensure that each modality retains its unique "flavor."
Summary updated