TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability

Fengji Ma, Li Liu, Hei Victor Cheng

Abstract

Vision-Language Models (VLMs) like CLIP have transformed computer vision through their zero-shot generalization capabilities. However, these models remain highly vulnerable to adversarial perturbations. Existing defense mechanisms often suffer from a trade-off where increasing robustness leads to a significant decline in clean image accuracy and generalization. We propose TIMA (Text-Image Mutual Awareness), a fine-tuning framework that addresses this imbalance. By maximizing inter-class distances in both text and image embedding spaces while preserving semantic integrity through cross-modal distillation, TIMA achieves state-of-the-art adversarial robustness without sacrificing zero-shot performance.

Introduction

Large-scale VLMs like CLIP allow for "zero-shot" classification, identifying objects in categories not explicitly seen during training. Despite this flexibility, they possess a "glass jaw": human-imperceptible adversarial noise can easily derail their predictions. Current defenses face a "Goldilocks" problem: methods like TeCoA perform well against minor noise but fail against larger perturbations, while others like LAAT improve robustness at the cost of "breaking" the model's semantic common sense on clean data. TIMA seeks a balanced solution by making the model resilient to attacks while maintaining its inherent intelligence.

Methodology: Mutual Awareness

TIMA balances robustness and generalization through two primary "Mutual Awareness" modules that refine both text and image modalities.

1. Image-Aware Text (IAT) Tuning

To enhance class distinguishability, TIMA employs Minimum Hyperspherical Energy (MHE). This technique treats text embeddings as points on a sphere that naturally repel one another, maximizing inter-class distance. To prevent these shifted embeddings from losing their semantic meaning, Image-Aware Knowledge Distillation (IAKD) is used. Here, fixed pre-trained image embeddings act as a "teacher," ensuring the new text embeddings remain semantically grounded.

2. Text-Aware Image (TAI) Tuning

Unlike previous methods that focused solely on text, TIMA optimizes the image embedding space. It introduces a Text-distance Adaptive Margin (TAM). For semantically similar classes (e.g., "cat" and "lynx"), a larger margin is applied during training to force their image embeddings further apart. Complementing this, Text-Aware Knowledge Distillation (TAKD) keeps the fine-tuned image embeddings aligned with the original pre-trained text space.

Confusion matrices comparing CLIP, LAAT, and TIMA (Ours), demonstrating improved class separation in TIMA.

As shown in the confusion matrices above, TIMA maintains much cleaner class separation compared to standard CLIP and the LAAT baseline, particularly for challenging datasets like ImageNet.

Experimental Results

TIMA was fine-tuned on ImageNet and Tiny-ImageNet and evaluated across 10 diverse zero-shot datasets, including CIFAR10, Food101, and SUN397.

Robustness and Generalization

TIMA consistently outperformed state-of-the-art methods (TeCoA, PMG, and LAAT) across nearly all benchmarks:

  • Clean Accuracy: TIMA achieved an average of 64.37%, significantly outperforming LAAT (51.60%) and TeCoA (59.85%).
  • Small-Scale Attacks ($\epsilon = 1/255$): TIMA reached 46.41% average robust accuracy, surpassing the next best method, PMG, by 2.64%.
  • Large-Scale Attacks ($\epsilon = 8/255$): TIMA demonstrated its greatest strength here, maintaining 22.64% accuracy, while competitors like TeCoA and PMG collapsed to 7.87% and 4.66%, respectively.
Radar charts showing TIMA's superior performance across multiple datasets and attack scenarios compared to baselines.

The radar charts illustrate TIMA's (red line) consistent dominance across various datasets and attack intensities, maintaining a larger area of performance than competing methods.

Conclusion

TIMA demonstrates that increasing inter-class distance in both image and text modalities is essential for defending against adversarial attacks. By utilizing cross-modal distillation, the model retains the "latent knowledge" of the original CLIP while becoming significantly more robust. This dual-modality approach provides a robust framework for securing large-scale Vision-Language Models without compromising their hallmark generalization abilities.

Summary updated

BibTeX
@inproceedings{ma2026tima,
  author = {Fengji Ma and Li Liu and Hei Victor Cheng},
  title = {{TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i29.39603},
}
Figure