Beyond Attack-Based Accuracy: Rethinking Adversarial Robustness Evaluation Through Structural Gradient Alignment

Hanrui Wang, Yu Luo, Shuo Wang, Xingbo Dong, Zhe Jin, Isao EchizenarXiv 2026 · arXiv

TL;DR

Structural Gradient Alignment (SGA) is a novel, attack-agnostic metric that evaluates adversarial robustness by measuring how well a model's local sensitivity aligns with the spatial structure of input images. It enables efficient, few-shot auditing of diverse vision systems—from standard CNNs to large vision-language models—without requiring labeled data, specific attack protocols, or heavy computation.

What Problem Does This Paper Address?

Traditionally, we measure a model's "robustness" by its attack-based accuracy: we throw a specific adversarial attack (like PGD) at it and see how often it still gets the right answer. While effective, this paradigm has four major flaws:

  1. Attack Dependence: Results change depending on which attack or "threat norm" you choose.
  2. Data Dependence: It usually requires labeled, in-distribution test sets, which might be private or unavailable to external auditors.
  3. Computational Cost: Iterative attacks are slow, especially for massive models.
  4. Model Evolution: Traditional metrics don't easily translate to modern Vision-Language Models (VLMs) and autoregressive Large Vision-Language Models (LVLMs) that don't just output a single class label.

The authors seek a "first-stage" evaluation tool that can rank models by robustness quickly and with minimal information.

Methods

The core of the paper is the structural sensitivity hypothesis: robust models tend to align their local input sensitivity with stable spatial structures (like edges and shapes), whereas weaker models exhibit fragmented or incidental sensitivity.

To measure this, the authors developed the SGA pipeline:

  • Objective Construction: Instead of needing a ground-truth label, SGA uses the model's own prediction as a "pseudo-target" to compute a gradient.
  • Gradient Extraction: It performs exactly one backward pass to find the input gradient.
  • Structural Mapping: Both the original image and the gradient are transformed into binary "structural maps" (essentially edge detection).
  • Alignment Scoring: It calculates the similarity (using Dice or SSIM) between the image's edges and the gradient's "edges."
Figure 2. SGA scoring pipeline. For each model–probe pair, SGA transforms the probe image and its input gradient into structural maps under configuration c = (ρ, ψ, e, µ). It then measures their structural correspondence and aggregates the resulting scores across the probe batch to rank the evaluated models.

The authors propose three protocols: Transfer SGA (uses a fixed configuration), Uncalibrated SGA (self-optimizes for the best ranking signal), and Calibrated SGA (uses a few "anchor" models with known robustness to set the scale).

Datasets

The authors evaluated SGA across 38 distinct models, including standard Deep Neural Networks (DNNs), VLMs, and LVLMs. They used four primary data sources for their "probes":

  • In-distribution: NIPS development set (ImageNet-1K) and VQAv2.
  • Task-independent "Public Probes": CelebA-HQ (faces) and Places365 (scenes).

Crucially, they showed that you can evaluate an ImageNet classifier's robustness using random pictures of faces, proving the method is remarkably data-agnostic.

Key Insights and Ideas

  • Robustness is Visually Apparent: As shown in Figure 1, the "sensitivity structure" of a highly robust model looks like a clean sketch of the input image. In contrast, a non-robust model's sensitivity looks like random noise.
  • One Image is Enough: SGA is incredibly efficient. While more images help, the method provides a meaningful robustness signal even with a single random probe image ($K=1$).
  • Architecture Matters: The study found that CNNs (like ResNet) naturally exhibit more continuous sensitivity patterns than Vision Transformers (ViT), which tend to have more fragmented, patch-wise sensitivity. This suggests that the "locality" of CNNs might be inherently linked to certain types of structural robustness.
  • Attack-Agnosticism: You don't need to simulate an attack to know if a model is vulnerable. If its gradients don't "make sense" structurally, it's likely fragile.
Figure 1. Illustration of SGA across robustness levels and probe sources. SGA measures structural alignment between input structure and objective- conditioned local sensitivity. Across test-set and task-independent public probes, higher-reference-robustness models exhibit stronger alignment and higher SGA scores. Reference ℓ∞PGD robust accuracies are shown only for comparison and are not used by SGA.

Key Results

  • High Ranking Agreement: Uncalibrated SGA achieved an average 83.72% pairwise agreement with gold-standard PGD rankings across various model types. It outperformed the strongest baseline by an average of 9.38 percentage points.
  • Massive Speedup: SGA can rank five Large Vision-Language Models in just 103.8 seconds on a single H100 GPU. For comparison, traditional AutoAttack can take hours for a single model.
  • Cross-Model Consistency: SGA is the only evaluated method that consistently ranked both simple classifiers and complex LVLMs correctly using the same task-independent probes.

Limitations and Future Work

  • Within-Pool Ranking: Currently, SGA is best at ranking models that share a "homogeneous pool" (e.g., all $\ell_\infty$ models for the same task). It isn't yet designed to compare an $\ell_2$ model directly against an $\ell_\infty$ model.
  • Theoretical Foundation: While the empirical evidence for the "structural sensitivity hypothesis" is strong, a formal theoretical proof linking this alignment to adversarial robustness remains an open challenge.
  • Future Directions: The authors suggest SGA could be used for "checkpoint triage"—quickly scanning hundreds of training checkpoints to find the most robust ones before committing to expensive full-scale testing. It could also potentially be used as a training objective to encourage models to develop more robust internal representations.

Summary updated

Abstract

Visual adversarial robustness remains a fundamental security property as vision systems evolve from conventional classifiers to vision-language models (VLMs) and large vision-language models (LVLMs), with visual inputs remaining an attack surface across generations. Yet existing evaluation methods often depend on prescribed attacks, in-distribution data, extensive probing, or classifier-specific formulations, limiting external auditing of diverse models. We introduce Structural Gradient Alignment (SGA), a complementary robustness metric measuring the structural alignment between input content and objective-conditioned local model sensitivity. SGA uses a few randomly sampled, unlabeled public probes that need not follow the downstream task distribution and only one input-gradient computation per model--probe pair. Across 38 distinct DNNs, VLMs, and LVLMs, SGA produces rankings consistent with attack-based references. Without robustness references, uncalibrated SGA achieves $83.72\%$ average pairwise agreement across eight public-probe conditions, outperforming the strongest overall baseline in all eight by $9.38$ percentage points on average. With known robustness orders of anchor models, calibrated SGA increases agreement to $85.36\%$. These results demonstrate an attack-agnostic, computationally efficient robustness signal across conventional DNNs, VLMs, and autoregressive LVLMs, downstream tasks, threat norms, probe distributions, and evaluator-knowledge settings. Our implementation is available at https://github.com/azrealwang/SGA.
BibTeX
@misc{wang2026beyond,
  author = {Hanrui Wang and Yu Luo and Shuo Wang and Xingbo Dong and Zhe Jin and Isao Echizen},
  title = {{Beyond Attack-Based Accuracy: Rethinking Adversarial Robustness Evaluation Through Structural Gradient Alignment}},
  year = {2026},
  eprint = {2611.17377},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17377},
}
Figure