Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-Grained Knowledge Alignment

Alvi Md. Ishmam, Christopher ThomasCVPR 2024 · DOI · arXiv

Introduction

Vision-Language Models (VLMs) like CLIP have transformed artificial intelligence by enabling models to learn from vast, web-scale datasets. However, this reliance on uncurated data introduces a critical vulnerability: it is remarkably easy for an adversary to inject poisoned samples into the training pool. These attacks typically manifest as backdoors, where a specific trigger (e.g., a checkerboard patch) causes the model to misclassify an image, or poisoning, which shifts the model's overall behavior through mislabeled pairs.

Overview of the Semantic Shield defense mechanism against adversarial poisoning.

Existing defenses often fail because they require clean datasets—negating the benefits of web-scale training—or they significantly degrade performance on legitimate data. Semantic Shield addresses this by forcing models to "justify" their attention through fine-grained semantic grounding, ensuring they ignore malicious triggers that lack physical meaning.

Methodology: Fine-Grained Knowledge Alignment

Semantic Shield operates on the principle that while an attack trigger might be statistically correlated with a target label, it lacks the semantic attributes inherent to that category. The framework consists of four primary components designed to enforce this grounding during training:

  1. Knowledge Element (KE) Generation: For every training caption, a Large Language Model (Vicuna) generates a list of constituent visual attributes. For instance, a "dog" caption might yield KEs like "paws," "furry coat," or "sharp teeth."
  2. Patch-KE Alignment ($L_{KE}$): This objective function forces the model to align specific image patches with these KEs. Because a random noise patch or backdoor trigger does not consistently align with semantic KEs across different images, the model learns to treat the trigger as noise and the KEs as the primary signal.
  3. KE-Guided Attention ($L_{Attention}$): The Vision Transformer (ViT) training is modified to penalize attention paid to patches that lack alignment with the KEs. If the model attempts to focus on a backdoor patch to identify a target class, the system suppresses that attention because the patch contains no relevant semantic attributes.
  4. Dynamic Sample Weighting: The model automatically downweights training samples where the overall image-to-KE alignment is low, effectively silencing poisoned samples during the learning process.
The detailed architecture of Semantic Shield, illustrating the flow from KE generation to the final alignment and attention losses.

Crucially, this defense is integrated entirely into the training phase. The resulting model is a standard VLM that requires no extra computation or LLM calls during inference.

Experimental Results

The researchers evaluated Semantic Shield on standard benchmarks, including MS-COCO and Flickr30k. The results demonstrate a dramatic reduction in Attack Success Rate (ASR) while preserving model utility:

  • Backdoor Defense: On the COCO dataset, an undefended model had a Hit@1 (attacker success rate) of 90.66% for patch attacks. Semantic Shield reduced this to a mere 0.9%.
  • Stealthy Attacks: Against sophisticated "imperceptible noise" attacks like BPP and Wanet, the defense reduced the attacker's success rate from 100% to 0%.
  • Utility Preservation: Unlike many defensive techniques that degrade general performance, Semantic Shield maintained image-text retrieval accuracy nearly identical to clean, undefended models.

Conclusion

Semantic Shield demonstrates that semantic grounding can serve as a powerful physical constraint against adversarial manipulation. By treating common-sense visual knowledge as a filter, the model naturally ignores spurious correlations introduced by attackers. While the method's effectiveness is tied to the quality of the LLM-generated KEs, it represents a significant step toward securing web-scale VLM training without sacrificing performance.

Summary updated

BibTeX
@inproceedings{ishmam2024semantic,
  author = {Alvi Md. Ishmam and Christopher Thomas},
  title = {{Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-Grained Knowledge Alignment}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02344},
}
Figure