Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking

Zefeng Zhang, Jiawei Sheng, Chuang Zhang, Liangyunzhi Liangyunzhi, Wenyuan Zhang, Siqi Wang, Tingwen Liu

TL;DR

The paper introduces OT-MEL, a framework that uses Optimal Transport (OT) to improve Multimodal Entity Linking. By formulating the alignment between text tokens and image patches as a global optimization problem, the model avoids the "greedy" pitfalls of standard attention mechanisms, leading to more accurate entity disambiguation in complex, multi-object scenes.

What Problem Does This Paper Address?

Multimodal Entity Linking (MEL) is the task of connecting ambiguous mentions in text (like the name "Robert") to the correct entity in a Knowledge Graph (KG) using both textual and visual context.

The core challenge is the modality gap: how do you effectively correlate a word in a sentence with a specific patch of an image? Most current models rely on standard attention mechanisms. However, attention often suffers from "degraded dominant correspondence"—it might over-focus on easy, local matches (like a name) while ignoring subtle but crucial global cues (like the specific facial features of an actor vs. a politician). This makes it difficult to distinguish between entities that share similar names or contexts.

Multimodal Entity Linking (MEL) task: disambiguating the mention 'Robert' by leveraging visual context to select the correct entity from a Knowledge Graph.

Methods

The authors propose OT-MEL, which replaces or guides traditional attention with Optimal Transport.

  1. Feature Extraction: The model uses a CLIP-based architecture, employing BERT for textual embeddings and a Vision Transformer (ViT) for visual patches.
  2. OT-based Correlation Assignment: Instead of calculating independent attention weights, the model treats the alignment of $n$ text tokens and $m$ image patches as a "transportation" problem. It seeks a global plan that minimizes the total "cost" of moving information between modalities, where cost is defined by the cosine distance between features. This is solved efficiently using the Sinkhorn-Knopp algorithm.
  3. Cross-Modal and Unimodal Matching: OT is applied twice: first to fuse text and image features (multimodal interaction) and second to match the resulting mention representation against candidate entities (fine-grained matching).
  4. Knowledge Distillation (KD): Because the Sinkhorn-Knopp algorithm can be computationally expensive during inference, the authors use KD to "teach" a standard attention module to mimic the global transport plan. This allows the model to maintain OT-level accuracy with the speed of a simple attention mechanism.
The overall OT-MEL framework, showing multimodal feature extraction and the OT-based correlation assignment modules.

Datasets

The model was evaluated on three widely-used MEL benchmarks:

  • RichpediaMEL: ~17,000 samples, primarily focused on person entities.
  • WikiMEL: ~25,000 samples collected from Wikipedia.
  • WikiDiverse: ~15,000 samples covering diverse topics like sports, technology, and economy.

Key Insights and Ideas

  • Global vs. Local: Standard attention is "greedy" and local. OT is "global" because it considers the entire distribution of tokens and patches simultaneously, ensuring that every element is appropriately accounted for in the final matching score.
  • The Power of Entropy: The authors found that the entropy regularization parameter ($\lambda$) in the OT formulation acts as a crucial "focus" dial. An appropriate level of entropy prevents the model from being either too concentrated on a single point or too blurry across the whole image.
  • Robustness in Complexity: OT-MEL shines particularly in "multi-object" images. When an image contains multiple people or objects, standard attention often gets confused by distractors. OT's global constraints help it stay locked onto the correct visual evidence.

Key Results

OT-MEL achieved state-of-the-art results across all three datasets:

  • Accuracy Gains: On the RichpediaMEL dataset, OT-MEL achieved an 83.30% H@1 score, outperforming the previous best model (MIMIC) by over 2%.
  • Multi-Object Performance: In scenarios with multiple objects in the image, OT-MEL showed a massive 10.40% improvement in H@1 compared to attention-based baselines.
  • Efficiency: The Knowledge Distillation (KD) version of the model achieved nearly identical performance to the full OT version while significantly reducing inference time (e.g., a 337-second speedup on the RichpediaMEL validation set).

Limitations and Future Work

  • Uniform Distribution Assumption: Currently, the model assumes that all tokens and patches carry equal "mass" (importance). In reality, some words or image regions are more informative than others. Exploring non-uniform distributions is a suggested next step.
  • Modality Expansion: The current framework is tailored for text and static images. The authors note that extending this OT-guided approach to video and speech modalities remains an open research direction.
  • Generative Linking: While this paper focuses on discriminative linking (choosing from a list), the authors suggest that OT could also improve generative entity linking models, which currently struggle with lexical ambiguity.

Summary updated

Abstract

Multimodal entity linking (MEL) aims to link ambiguous mentions in multimodal contexts to entities in a multimodal knowledge graph. A pivotal challenge is to fully leverage multi-element correlations between mentions and entities to bridge modality gap and enable fine-grained semantic matching. Existing methods attempt several local correlative mechanisms, relying heavily on the automatically learned attention weights, which may over-concentrate on partial correlations. To mitigate this issue, we formulate the correlation assignment problem as an optimal transport (OT) problem, and propose a novel MEL framework, namely OT-MEL, with OT-guided correlation assignment. Thereby, we exploit the correlation between multimodal features to enhance multimodal fusion, and the correlation between mentions and entities to enhance fine-grained matching. To accelerate model prediction, we further leverage knowledge distillation to transfer OT assignment knowledge to attention mechanism. Experimental results show that our model significantly outperforms previous state-of-the-art baselines and confirm the effectiveness of the OT-guided correlation assignment.
BibTeX
@inproceedings{zhang2024optimal,
  author = {Zefeng Zhang and Jiawei Sheng and Chuang Zhang and Liangyunzhi Liangyunzhi and Wenyuan Zhang and Siqi Wang and Tingwen Liu},
  title = {{Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.243},
}
Figure