TL;DR
Adaptive Geodesic Correction (AGC) is a training-free, test-time defense that dramatically boosts the adversarial robustness of Vision-Language Models like CLIP. By treating specific data augmentations as geometric anchors on a hypersphere, AGC corrects corrupted features in a single forward pass, achieving a 10$\times$ speedup and over 40% higher robust accuracy compared to current state-of-the-art gradient-based defenses.
What Problem Does This Paper Address?
Vision-Language Models (VLMs), particularly CLIP, have revolutionized zero-shot image classification. However, they share a common Achilles' heel with traditional computer vision models: adversarial vulnerability. Small, imperceptible perturbations to an image can flip a model's prediction from "Great Pyrenees" to "Akbash" with high confidence.
While researchers have developed "test-time defenses" to fix models on the fly without retraining, these methods usually rely on gradient-based optimization (backpropagation) during inference. This makes them computationally expensive and slow—often too slow for real-world deployment. Furthermore, while many methods use data augmentation, they often do so indiscriminately, failing to exploit the specific geometric properties of how different augmentations affect a model's feature space.
Methods
The authors propose Adaptive Geodesic Correction (AGC), a three-step pipeline that operates entirely during the inference (forward) pass:
- Reliable Augmentation Selection: The authors discovered that not all augmentations are created equal. By measuring the "classification margin"—the difference in similarity between the correct class and the next best guess—they found that certain augmentations like
RandomPerspectiveconsistently steer adversarial features back toward the correct semantics. - Geodesic Correction: Since CLIP features are $\ell_2$-normalized, they live on a unit hypersphere. AGC identifies a "geometric anchor" by averaging several views of the input image under the selected reliable augmentation. It then moves the original input feature along the shortest path (the geodesic) toward this anchor.
- Adaptive Step Size: A critical challenge is distinguishing clean images from adversarial ones at test time. AGC uses two signals—feature deviation (how far the input is from its augmented versions) and anchor reliability (how consistent the augmented versions are with each other)—to calculate an adaptive step size. If the input looks adversarial, the correction is aggressive; if it looks clean, the correction is conservative to preserve original accuracy.
Datasets
The authors evaluated AGC across eight diverse fine-grained classification datasets:
- General Objects: Caltech101
- Animals & Plants: OxfordPets, Flower102
- Vehicles: StanfordCars, FGVCAircraft
- Textures: DTD
- Satellite Imagery: EuroSAT
- Human Actions: UCF101
Testing was performed across three CLIP backbones (ViT-B/32, ViT-B/16, and ViT-L/14) and ResNet50.
Key Insights and Ideas
- Geometry Matters: The paper's most significant "aha moment" is the strong positive correlation between an augmentation's ability to improve the classification margin and its resulting adversarial robustness. As shown in Figure 1, augmentations like
PerspectiveandRandom Cropare far more effective thanSolarizeorHorizontal Flip. - Augmentations as Semantic Compasses: Adversarial attacks work by pushing features away from their true semantic cluster. AGC shows that robust augmentations act as a compass, providing a reliable direction to "pull" those features back home.
- Efficiency via Forward Passes: By eliminating the need for backpropagation at test time, the authors prove that robustness doesn't have to come at the cost of massive latency.
Key Results
- Superior Robustness: On the ViT-B/32 backbone, AGC achieved an average robust accuracy of 82.9%, a massive leap over the previous state-of-the-art (TTP at 39.7% and R-TPT at 35.3%).
- High-End Performance: With the larger ViT-L/14 backbone, AGC reached a staggering 93.4% average robust accuracy.
- Speed: AGC is roughly 10$\times$ faster than TTP and 29$\times$ faster than R-TPT on clean samples, taking only 0.0091 seconds per sample on an NVIDIA A100 GPU.
- Clean Accuracy Preservation: Unlike many defenses that degrade performance on normal images, AGC maintains clean accuracy comparable to the original "vanilla" CLIP model.
Limitations and Future Work
While highly effective, AGC currently relies on a set of predefined hyperparameters (like the choice of RandomPerspective and the trade-off coefficient $\gamma$). The authors note that while these parameters generalize well across the datasets they tested, a truly "hyperparameter-free" and "update-free" defense remains an open challenge.
Future work could involve automatically discovering optimal augmentation strategies for different model architectures or developing even more sophisticated adaptive mechanisms that don't require manual tuning.
Summary updated