AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models

Zhiwei Li, Jiacheng Xue, Weining Wang, Ajian Liu, Xingyu Gao, Zhenan Sun, Qi LiarXiv 2026 · arXiv

TL;DR

Adaptive Geodesic Correction (AGC) is a training-free, test-time defense that dramatically boosts the adversarial robustness of Vision-Language Models like CLIP. By treating specific data augmentations as geometric anchors on a hypersphere, AGC corrects corrupted features in a single forward pass, achieving a 10$\times$ speedup and over 40% higher robust accuracy compared to current state-of-the-art gradient-based defenses.

What Problem Does This Paper Address?

Vision-Language Models (VLMs), particularly CLIP, have revolutionized zero-shot image classification. However, they share a common Achilles' heel with traditional computer vision models: adversarial vulnerability. Small, imperceptible perturbations to an image can flip a model's prediction from "Great Pyrenees" to "Akbash" with high confidence.

While researchers have developed "test-time defenses" to fix models on the fly without retraining, these methods usually rely on gradient-based optimization (backpropagation) during inference. This makes them computationally expensive and slow—often too slow for real-world deployment. Furthermore, while many methods use data augmentation, they often do so indiscriminately, failing to exploit the specific geometric properties of how different augmentations affect a model's feature space.

Methods

The authors propose Adaptive Geodesic Correction (AGC), a three-step pipeline that operates entirely during the inference (forward) pass:

  1. Reliable Augmentation Selection: The authors discovered that not all augmentations are created equal. By measuring the "classification margin"—the difference in similarity between the correct class and the next best guess—they found that certain augmentations like RandomPerspective consistently steer adversarial features back toward the correct semantics.
  2. Geodesic Correction: Since CLIP features are $\ell_2$-normalized, they live on a unit hypersphere. AGC identifies a "geometric anchor" by averaging several views of the input image under the selected reliable augmentation. It then moves the original input feature along the shortest path (the geodesic) toward this anchor.
  3. Adaptive Step Size: A critical challenge is distinguishing clean images from adversarial ones at test time. AGC uses two signals—feature deviation (how far the input is from its augmented versions) and anchor reliability (how consistent the augmented versions are with each other)—to calculate an adaptive step size. If the input looks adversarial, the correction is aggressive; if it looks clean, the correction is conservative to preserve original accuracy.
Figure 2: Overview of Adaptive Geodesic Correction (AGC). AGC first selects a reliable augmentation anchor using the margin criterion. The original feature is then corrected along the geodesic toward this anchor on CLIP’s unit hypersphere. The correction step size is adaptively determined by feature deviation and anchor reliability, allowing AGC to improve adversarial robustness while preserving clean-sample semantics, without requiring backpropagation-based test-time updates.

Datasets

The authors evaluated AGC across eight diverse fine-grained classification datasets:

  • General Objects: Caltech101
  • Animals & Plants: OxfordPets, Flower102
  • Vehicles: StanfordCars, FGVCAircraft
  • Textures: DTD
  • Satellite Imagery: EuroSAT
  • Human Actions: UCF101

Testing was performed across three CLIP backbones (ViT-B/32, ViT-B/16, and ViT-L/14) and ResNet50.

Key Insights and Ideas

  • Geometry Matters: The paper's most significant "aha moment" is the strong positive correlation between an augmentation's ability to improve the classification margin and its resulting adversarial robustness. As shown in Figure 1, augmentations like Perspective and Random Crop are far more effective than Solarize or Horizontal Flip.
  • Augmentations as Semantic Compasses: Adversarial attacks work by pushing features away from their true semantic cluster. AGC shows that robust augmentations act as a compass, providing a reliable direction to "pull" those features back home.
  • Efficiency via Forward Passes: By eliminating the need for backpropagation at test time, the authors prove that robustness doesn't have to come at the cost of massive latency.
Figure 1: Relationship between first-order derivation and robustness using different augmentations across three CLIP backbones.

Key Results

  • Superior Robustness: On the ViT-B/32 backbone, AGC achieved an average robust accuracy of 82.9%, a massive leap over the previous state-of-the-art (TTP at 39.7% and R-TPT at 35.3%).
  • High-End Performance: With the larger ViT-L/14 backbone, AGC reached a staggering 93.4% average robust accuracy.
  • Speed: AGC is roughly 10$\times$ faster than TTP and 29$\times$ faster than R-TPT on clean samples, taking only 0.0091 seconds per sample on an NVIDIA A100 GPU.
  • Clean Accuracy Preservation: Unlike many defenses that degrade performance on normal images, AGC maintains clean accuracy comparable to the original "vanilla" CLIP model.

Limitations and Future Work

While highly effective, AGC currently relies on a set of predefined hyperparameters (like the choice of RandomPerspective and the trade-off coefficient $\gamma$). The authors note that while these parameters generalize well across the datasets they tested, a truly "hyperparameter-free" and "update-free" defense remains an open challenge.

Future work could involve automatically discovering optimal augmentation strategies for different model architectures or developing even more sophisticated adaptive mechanisms that don't require manual tuning.

Summary updated

Abstract

Vision-language models like CLIP have demonstrated remarkable zero-shot transfer capabilities. However, their susceptibility to imperceptible adversarial perturbations remains a critical security concern. While test-time defenses offer a pragmatic solution for deployed models, existing approaches typically rely on gradient-based optimization during inference, incurring significant computational overhead. In this paper, we revisit the role of data augmentation in CLIP robustness and observe that augmentations are not equally effective: specific augmentations consistently provide robust geometric cues that align with correct class semantics in the hyperspherical feature space. Based on this, we propose Adaptive Geodesic Correction (AGC), a training-free defense mechanism that requires no parameter updates. AGC identifies a reliable augmentation as a geometric anchor and corrects the input feature towards it, utilizing an adaptive step size to balance robustness against clean accuracy preservation. AGC achieves superior performance across eight fine-grained datasets and three CLIP backbones, improving average robust accuracy by 44.4\% over state-of-the-art baseline while delivering a 10$\times$ reduction in inference latency. Our findings reveal a fundamental geometric property of CLIP features, offering a highly efficient and effective paradigm for robust multimodal deployment.
BibTeX
@misc{li2026agc,
  author = {Zhiwei Li and Jiacheng Xue and Weining Wang and Ajian Liu and Xingyu Gao and Zhenan Sun and Qi Li},
  title = {{AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models}},
  year = {2026},
  eprint = {2605.15584},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.15584},
}

Loading related content...

Figure