Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models

Futa Waseda, Saku Sugawara, Isao Echizen

TL;DR

Vision-Language Models (VLMs) like CLIP are highly vulnerable to adversarial attacks, but current defense methods either overfit to simple class labels or ignore semantic text guidance entirely. This paper introduces Quality Text-guided Adversarial Fine-Tuning (QT-AFT), a method that uses rich, detailed image captions to guide adversarial training. By forcing the model to recognize diverse visual features under noise, QT-AFT achieves state-of-the-art zero-shot robustness across 16 datasets without sacrificing clean accuracy.

The Problem: The "Goldilocks" Trap of Adversarial Training

Pre-trained VLMs like CLIP are the backbone of modern multimodal AI, enabling zero-shot image classification. However, they are brittle; adding tiny, imperceptible noise (adversarial attacks) can completely flip predictions. To counter this, researchers use Adversarial Training (AT)—training models on noisy images to ignore perturbations. Existing AT methods for VLMs face a significant trade-off:

  1. Supervised AT (e.g., TeCoA): Uses simple class labels (e.g., "a photo of a dog"). This causes the model to overfit to specific training objects, degrading its ability to generalize to unseen zero-shot tasks.
  2. Unsupervised AT (e.g., FARE): Ignores text and attempts to keep noisy image embeddings close to clean ones. Lacking semantic guidance, it fails to capture diverse meanings within an image, leaving it vulnerable to text-guided attacks.

Methodology: Quality Text-guided Adversarial Fine-Tuning (QT-AFT)

QT-AFT addresses these issues by replacing short class labels with detailed, descriptive image captions during the adversarial fine-tuning of CLIP's vision encoder.

Overview of the QT-AFT framework, showing how detailed captions guide adversarial training through a dual-objective loss.

The method follows a two-step process:

  1. Caption Generation: A powerful VLM (InternVL-2.5-8B) generates detailed, ~50-word descriptions of training images before training begins.
  2. Dual-Objective Adversarial Training: Adversarial examples (AEs) are generated to maximize a combined loss function. The unsupervised component pushes the AE away from the original image embedding, while the supervised component pushes it away from the detailed caption's embedding.

By maximizing deviation from a rich caption (describing objects, colors, backgrounds, and relationships), the generated noise disrupts a wide array of semantic features. Training the model to resist this complex noise results in a more robust, holistic vision encoder.

Key Insights: What Makes Language Effective?

Ablation studies revealed critical findings regarding linguistic supervision:

  • Captions > Labels: Detailed captions prevent the severe zero-shot overfitting seen in label-based methods.
  • Descriptive Language is Vital: Removing adjectives and adverbs—which capture transferable attributes like color and shape—significantly drops robustness.
  • Relational Context Matters: Removing function words or shuffling word order degrades performance, indicating that the model relies on natural language structure rather than just a "bag of words."

Results and Performance

QT-AFT was evaluated on 16 diverse datasets, including ImageNet variants, fine-grained tasks (OxfordPets, Flowers102), and domain-specific tasks (EuroSAT, PCAM).

Zero-shot performance of QT-AFT compared to baseline methods across 16 diverse datasets, demonstrating superior robust accuracy.
  • State-of-the-Art Robustness: QT-AFT achieved the highest zero-shot robust accuracy (against AutoAttack) on 12 out of 16 datasets, with an average robust accuracy of 25.8% (outperforming FARE's 22.4% and TeCoA's 21.8%).
  • Maintaining Clean Accuracy: Unlike supervised methods that often suffer a "robustness tax," QT-AFT maintained a high average clean accuracy of 51.6%.
  • Transferability: The authors successfully transferred this robustness to Large Vision-Language Models (LVLMs) like LLaVA, improving performance on complex tasks like Visual Question Answering (VQA) under attack.

Conclusion

QT-AFT demonstrates that high-quality linguistic supervision is a powerful tool for building robust visual representations. While generating captions adds initial computational overhead and is currently limited by CLIP's 77-token text window, the results suggest a paradigm shift: moving away from unimodal adversarial training toward semantically rich, text-guided robustness.

Summary updated

BibTeX
@inproceedings{waseda2025quality,
  author = {Futa Waseda and Saku Sugawara and Isao Echizen},
  title = {{Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3755623},
}
Figure