Abstract
Multimodal Entity Linking (MEL) is a critical task for grounding visual and textual information in structured knowledge bases. However, the robustness of these models against adversarial perturbations—tiny, intentional changes to images—remains largely unexplored. This paper presents a systematic evaluation of visual adversarial attacks on MEL foundation models, revealing significant vulnerabilities. To address these, we introduce LLM-RetLink, a retrieval-augmented framework that leverages Large Language Models and dynamic context to significantly improve performance under adversarial conditions.
Introduction: The Vulnerability of Multimodal Linking
Multimodal Entity Linking (MEL) connects images and their associated text to specific entries in knowledge bases like Wikipedia. While foundational for AI "understanding," current MEL research often assumes clean input data. In reality, images can be maliciously altered. Adversarial attacks—imperceptible pixel changes—can trick models into incorrect associations, such as misidentifying an apple as a basketball. This study investigates the stability of MEL models when their visual inputs are compromised.
Methodology: Attacking and Defending MEL
The research evaluates both traditional encoders (CLIP, BLIP, ALIGN) and Multimodal Large Language Models (MLLMs like LLaVA).
Adversarial Attack Phase
The authors applied three standard gradient-based white-box attacks: PGD (Projected Gradient Descent), APGD (Auto-PGD), and CW (Carlini & Wagner). These were tested across "Normal" and "Strong" perturbation levels in two scenarios: Image-to-Text (I2T) and Image+Text-to-Text (IT2T).
The Defense: LLM-RetLink
To counter these attacks, the authors proposed LLM-RetLink, a two-stage framework:
- Visual Perception: A Large Vision Model (LVM) extracts initial descriptors from the image.
- Dynamic Retrieval: These descriptors query external knowledge bases (Wikidata/Wikipedia) for semantic evidence. This retrieved context is then processed by an LLM for final entity disambiguation, allowing "knowledge" to override "visual confusion."
Datasets and Benchmarks
The study utilized five large-scale MEL benchmarks: Wikidata-MEL, Richpedia-MEL, WikiDiverse, WIKIPerson, and $M^3EL$. Notably, the authors have released the first MEL adversarial example dataset to facilitate further research.
Key Insights
- High Visual Fragility: Traditional MEL models are surprisingly fragile; relying solely on images (I2T) led to accuracy drops of nearly 20% under attack.
- Textual Mitigation: Textual context acts as a powerful shield. In IT2T scenarios, the impact of visual attacks was significantly reduced.
- Dynamic Context is Crucial: Unlike static descriptions, dynamic retrieval—fetching context on the fly—provides a robust defense against targeted visual deception.
Results and Performance
LLM-RetLink improved MEL accuracy by 0.4% to 35.7% across datasets. For instance, under a "Strong" CW attack on Richpedia-MEL, CLIP's accuracy rose from 12.7% to 45.8% when enhanced with LLM-RetLink. The framework effectively narrows the performance gap between clean and adversarial images.
Conclusion
While retrieval-augmented LLMs offer a path toward robustness, the underlying visual encoders remain susceptible. Future work should focus on making cross-modal alignment inherently more resilient to noise, rather than relying solely on the language component to correct visual errors.
Summary updated