On Evaluating the Adversarial Robustness of Foundation Models for Multimodal Entity Linking

Fang Wang, Yongjie Wang, Zonghao Yang, Minghao Hu, Xiaoying BaiarXiv 2025 · DOI · arXiv

Abstract

Multimodal Entity Linking (MEL) is a critical task for grounding visual and textual information in structured knowledge bases. However, the robustness of these models against adversarial perturbations—tiny, intentional changes to images—remains largely unexplored. This paper presents a systematic evaluation of visual adversarial attacks on MEL foundation models, revealing significant vulnerabilities. To address these, we introduce LLM-RetLink, a retrieval-augmented framework that leverages Large Language Models and dynamic context to significantly improve performance under adversarial conditions.

Introduction: The Vulnerability of Multimodal Linking

Multimodal Entity Linking (MEL) connects images and their associated text to specific entries in knowledge bases like Wikipedia. While foundational for AI "understanding," current MEL research often assumes clean input data. In reality, images can be maliciously altered. Adversarial attacks—imperceptible pixel changes—can trick models into incorrect associations, such as misidentifying an apple as a basketball. This study investigates the stability of MEL models when their visual inputs are compromised.

Figure 1: An example of an adversarial attack causing a model to misidentify an apple as a basketball, and how textual queries can mitigate this.

Methodology: Attacking and Defending MEL

The research evaluates both traditional encoders (CLIP, BLIP, ALIGN) and Multimodal Large Language Models (MLLMs like LLaVA).

Adversarial Attack Phase

The authors applied three standard gradient-based white-box attacks: PGD (Projected Gradient Descent), APGD (Auto-PGD), and CW (Carlini & Wagner). These were tested across "Normal" and "Strong" perturbation levels in two scenarios: Image-to-Text (I2T) and Image+Text-to-Text (IT2T).

Figure 3: The proposed framework for generating adversarial attacks and evaluating the robustness of MEL models and LLM-based defenses.

To counter these attacks, the authors proposed LLM-RetLink, a two-stage framework:

  1. Visual Perception: A Large Vision Model (LVM) extracts initial descriptors from the image.
  2. Dynamic Retrieval: These descriptors query external knowledge bases (Wikidata/Wikipedia) for semantic evidence. This retrieved context is then processed by an LLM for final entity disambiguation, allowing "knowledge" to override "visual confusion."

Datasets and Benchmarks

The study utilized five large-scale MEL benchmarks: Wikidata-MEL, Richpedia-MEL, WikiDiverse, WIKIPerson, and $M^3EL$. Notably, the authors have released the first MEL adversarial example dataset to facilitate further research.

Key Insights

  • High Visual Fragility: Traditional MEL models are surprisingly fragile; relying solely on images (I2T) led to accuracy drops of nearly 20% under attack.
  • Textual Mitigation: Textual context acts as a powerful shield. In IT2T scenarios, the impact of visual attacks was significantly reduced.
  • Dynamic Context is Crucial: Unlike static descriptions, dynamic retrieval—fetching context on the fly—provides a robust defense against targeted visual deception.

Results and Performance

LLM-RetLink improved MEL accuracy by 0.4% to 35.7% across datasets. For instance, under a "Strong" CW attack on Richpedia-MEL, CLIP's accuracy rose from 12.7% to 45.8% when enhanced with LLM-RetLink. The framework effectively narrows the performance gap between clean and adversarial images.

Conclusion

While retrieval-augmented LLMs offer a path toward robustness, the underlying visual encoders remain susceptible. Future work should focus on making cross-modal alignment inherently more resilient to noise, rather than relying solely on the language component to correct visual errors.

Summary updated

Abstract

The explosive growth of multimodal data has driven the rapid development of multimodal entity linking (MEL) models. However, existing studies have not systematically investigated the impact of visual adversarial attacks on MEL models. We conduct the first comprehensive evaluation of the robustness of mainstream MEL models under different adversarial attack scenarios, covering two core tasks: Image-to-Text (I2T) and Image+Text-to-Text (IT2T). Experimental results show that current MEL models generally lack sufficient robustness against visual perturbations. Interestingly, contextual semantic information in input can partially mitigate the impact of adversarial perturbations. Based on this insight, we propose an LLM and Retrieval-Augmented Entity Linking (LLM-RetLink), which significantly improves the model's anti-interference ability through a two-stage process: first, extracting initial entity descriptions using large vision models (LVMs), and then dynamically generating candidate descriptive sentences via web-based retrieval. Experiments on five datasets demonstrate that LLM-RetLink improves the accuracy of MEL by 0.4%-35.7%, especially showing significant advantages under adversarial conditions. This research highlights a previously unexplored facet of MEL robustness, constructs and releases the first MEL adversarial example dataset, and sets the stage for future work aimed at strengthening the resilience of multimodal systems in adversarial environments.
BibTeX
@misc{wang2025evaluating,
  author = {Fang Wang and Yongjie Wang and Zonghao Yang and Minghao Hu and Xiaoying Bai},
  title = {{On Evaluating the Adversarial Robustness of Foundation Models for Multimodal Entity Linking}},
  year = {2025},
  eprint = {2508.15481},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.15481},
}
Figure