TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability

Fengji Ma, Hei Victor Cheng, Chenxing Li, Li Liu

TL;DR

TIMA (Text-Image Mutual Awareness) is a new framework that significantly boosts the adversarial robustness of CLIP-like models without destroying their ability to recognize new objects. By adaptively adjusting decision margins based on class similarity and ensuring text embeddings don't "drift" too far from their original meanings, TIMA achieves superior performance against both small and large adversarial attacks across 14 different datasets.

What Problem Does This Paper Address?

Foundation models like CLIP are famous for "zero-shot" learning—the ability to classify objects they’ve never seen during training just by reading a text description. However, these models have a glass jaw: they are incredibly vulnerable to adversarial attacks, where tiny, invisible-to-humans pixels are added to an image to trick the AI.

Existing fixes usually fall into two traps:

  1. The Robustness-Generalization Trade-off: Making a model robust often makes it worse at classifying normal ("clean") images.
  2. The Large Perturbation Gap: Models trained to handle small attacks often collapse entirely when the attacker uses a slightly larger "budget" of noise (perturbation radius).

The authors identify that current methods fail because they don't account for how similar classes (like "cat" vs. "dog") are naturally harder to defend than distinct ones, and they often scramble the model's internal "semantic map" while trying to toughen it up.

Methods

The authors propose the TIMA framework, which consists of two "mutually aware" modules:

  1. Text-Aware Image (TAI) Tuning with ASAM:
    • Standard training uses a fixed margin for all classes. TIMA introduces the Adaptive Semantic-Aware Margin (ASAM).
  2. It explicitly reduces the margin for classes that are semantically similar (e.g., "car" and "truck") and for samples that are inherently ambiguous. This "calibrates" the decision boundaries to be more resilient against large, unseen perturbations.
  3. Image-Aware Text (IAT) Tuning with SC-MHE:
    • To prevent the model from forgetting what words mean, they use Semantic Consistent Minimum Hyperspherical Energy (SC-MHE).
  4. This technique pushes text embeddings apart (making them easier to distinguish) while using a regularization term to ensure the new embeddings still behave like the original, pre-trained ones. This preserves the model's "common sense" generalization.

Datasets

The model was fine-tuned on Tiny-ImageNet and evaluated across 14 diverse datasets to test its zero-shot capabilities:

  • General Classification: CIFAR10, CIFAR100, STL10, Caltech101, Caltech256, ImageNet.
  • Fine-grained Classification: OxfordPets, Food101, Flowers, StanfordCars, FGVC (Aircraft).
  • Specialized Tasks: SUN397 (scenes), DTD (textures).

Key Insights and Ideas

  • The Stable Offset: The researchers discovered that the "logit margin" (the gap between the correct class and the next best guess) shifts by a predictable, stable amount when moving from small to large attacks. This suggests we can "pre-adjust" for large attacks during training on small ones.
  • Similarity is a Vulnerability: There is a strong negative correlation between how similar two classes are and how robust they are. If two classes are semantically close, an attacker needs much less effort to flip the model's prediction.
  • Semantic Drift is the Enemy: Previous methods that moved text embeddings around to improve robustness often "broke" the relationships between words. TIMA proves that keeping the original semantic structure is key to maintaining high accuracy on clean images.

Key Results

TIMA sets a new state-of-the-art (SOTA) for zero-shot adversarial robustness:

  • Superior Robustness: Under a large attack ($\epsilon = 8/255$), TIMA achieved an average robust accuracy of 21.65%, dwarfing the previous best method (LAAT) at 11.70% and TeCoA at 8.35%.
  • Better Generalization: Unlike other robust models that get "dumber" on clean images, TIMA improved clean accuracy by 5.5% over TeCoA and 11.5% over LAAT.
  • Emergent Robustness: Remarkably, even when trained only on clean data, TIMA’s ASAM module acted as a regularizer that provided better "accidental" robustness than some models specifically trained against attacks.

Limitations and Future Work

While TIMA performs exceptionally well, the paper notes:

  • Computational Cost: The authors provide a brief analysis of training costs in the appendix, but adversarial training is inherently more resource-intensive than standard fine-tuning.
  • Attack Diversity: The study focuses primarily on $L_\infty$ PGD attacks. While they tested against others like AutoAttack and BIM, exploring robustness against a wider variety of non-gradient-based or physical-world attacks remains a future direction.
  • Scaling: The experiments were conducted on CLIP-B/32; testing how these adaptive margins scale to even larger models (like ViT-L or ViT-G) would be a valuable next step.

Summary updated

Abstract

Achieving zero-shot adversarial robustness without sacrificing generalization remains challenging for foundation models such as CLIP, especially under large adversarial perturbations. Through empirical analyses, we identify three critical yet overlooked issues: (1) Logit margins exhibit a stable offset between small and large adversarial perturbations, suggesting that explicitly adjusting margins could improve robustness against unseen large perturbations. (2) A significant negative correlation exists between logit margin and inter-class semantic similarity, indicating that semantic structures are insufficiently leveraged by existing methods. (3) Existing methods for adjusting text embeddings disrupt the intrinsic semantic consistency established by pre-trained models, undermining generalization capability. Motivated by these findings, we propose a novel Text-Image Mutual Awareness (TIMA) framework, including a Text-Aware Image (TAI) tuning module with an Adaptive Semantic-Aware Margin (ASAM) to explicitly calibrate logit margins, and an Image-Aware Text (IAT) tuning module with Semantic Consistent Minimum Hyperspherical Energy (SC-MHE) to preserve semantic consistency. Comprehensive experiments validate that TIMA significantly outperforms existing approaches by effectively addressing the identified limitations.
BibTeX
@inproceedings{ma2026tima,
  author = {Fengji Ma and Hei Victor Cheng and Chenxing Li and Li Liu},
  title = {{TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i29.39603},
}
Figure