ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimization

Ronghao Zhang, Shuaicheng Niu, Qi Deng, Yanjie Dong, Jian Chen, Runhao Zeng

Deploying deep learning models in the real world is a constant battle against "distribution shift." A model trained on clean studio photos often stumbles when it encounters blurry, rainy, or low-light images from a smartphone. Test-Time Adaptation (TTA) is the standard fix: it allows a model to update itself on the fly using unlabeled data it sees during inference.

However, traditional TTA relies on backpropagation (BP). While BP is the engine of modern AI, it is a memory hog and requires a fully differentiable computation graph. This makes it nearly impossible to run on tiny edge devices or on "quantized" models (which use low-precision math to save power).

In their recent paper, Zhang et al. introduce ZOTTA, a framework that achieves state-of-the-art adaptation using only forward passes. By ditching backpropagation in favor of Zeroth-Order Optimization (ZOO), ZOTTA opens the door for robust AI on hardware where it was previously impossible.

What Problem Does This Paper Address?

The core challenge is the "BP-dependency" of current adaptation methods. Backpropagation requires storing intermediate "activations" to calculate gradients, which can inflate memory usage by 5x to 10x compared to a simple forward pass. Furthermore, many efficient deployment techniques—like 8-bit quantization—create non-differentiable "steps" in the model that break backpropagation entirely.

Zeroth-Order Optimization (ZOO) is a potential solution. It estimates gradients by slightly "jiggling" the model's weights, running a forward pass, and seeing how the loss changes. It’s gradient-free and memory-efficient.

The catch? Naive ZOO is notoriously slow and unstable. In high-dimensional spaces (like a neural network with millions of parameters), the "jiggling" becomes noisy, and without ground-truth labels, the optimization often wanders off into the weeds.

Methods: Making Gradient-Free Optimization Practical

ZOTTA makes ZOO viable for TTA through two clever architectural "pruning" and "stabilizing" mechanisms:

  1. Distribution-Robust Layer Selection (DRLS): ZOO struggles when asked to optimize too many parameters at once. ZOTTA observes that not every layer needs to change to handle a domain shift. They use a "clustering purity" metric—running a small set of source and target data through the model and seeing which layers can still tell the domains apart. If a layer’s features are already "domain-invariant" (mixed together), ZOTTA freezes it. By only updating the "domain-sensitive" layers, they drastically reduce the dimensionality of the optimization problem.

  2. Spatial Feature Aggregation Alignment (SFAA): Most TTA methods try to minimize "entropy" (making the model more confident). But under heavy noise, a model's confidence is a lying signal. ZOTTA instead aligns the global statistics (mean and variance) of spatial features between the source domain and the test data. This provides a much smoother, more stable objective for the ZOO estimator to follow, reducing the variance of the estimated gradients.

Datasets

The authors evaluated ZOTTA across a rigorous battery of benchmarks:

  • ImageNet-C/R/Sketch/A: The "gold standard" for testing robustness against corruptions, artistic renditions, sketches, and adversarial examples.
  • CIFAR100-C: A smaller-scale benchmark for rapid iteration.
  • MathVista: A multimodal benchmark used to test ZOTTA’s scalability on a 3-billion parameter Large Multimodal Model (Qwen2.5-VL-3B).

Key Insights and Ideas

  • Dimensionality is the Enemy: The variance of a ZOO gradient estimate scales with the square root of the number of parameters ($d$). By using DRLS to prune the update space, ZOTTA moves from an impossible optimization problem to a manageable one.
  • Middle Layers Matter Most: The paper finds that shallow layers (capturing edges/textures) and very deep layers (collapsing features for classification) are often robust. The "middle" layers are the sweet spot for domain adaptation.
  • Stability Over Confidence: In a gradient-free setting, the "shape" of the loss landscape matters immensely. Feature alignment (SFAA) creates a "bowl-shaped" objective that is much easier for ZOO to navigate than the "spiky" landscape of entropy minimization.

Key Results

  • Massive Memory Savings: On a ViT-Base model, ZOTTA used only 825 MB of memory, compared to 5,165 MB for the popular BP-based method TENT—an 84% reduction.
  • Superior Accuracy: On ImageNet-C, ZOTTA achieved 66.6% accuracy, outperforming the best previous BP-free method (FOA, 62.4%) and even beating some BP-based methods like SAR (62.7%).
  • Quantization Friendly: ZOTTA successfully adapted an 8-bit quantized model to 64.8% accuracy, a scenario where backpropagation-based methods simply cannot function.
  • Architecture Agnostic: Unlike previous BP-free methods that only worked on specific architectures (like Vision Transformers), ZOTTA works equally well on CNNs (ResNet) and large multimodal models.

Limitations and Future Work

While ZOTTA is a major step forward, the authors and the methodology suggest a few areas for further exploration:

  • Forward Pass Latency: While ZOTTA saves memory, it requires multiple forward passes (typically 10) per adaptation step to estimate the gradient. In latency-critical applications, this "compute-for-memory" trade-off needs careful balancing.
  • Source Data Access: The method requires a small set of source domain samples (or pre-computed statistics) to calculate the initial alignment targets. In strictly "source-free" scenarios, this might require synthetic data generation.
  • Hyperparameter Sensitivity: While the paper shows robustness, the "purity threshold" for layer selection might still need tuning for vastly different model architectures.

Summary updated

Abstract

Test-time adaptation (TTA) aims to improve model robustness under distribution shifts by adapting to unlabeled test data, but most existing methods rely on backpropagation (BP), which is computationally costly and incompatible with non-differentiable models such as quantized models, limiting practical deployment on numerous edge devices. Recent BP-free approaches alleviate overhead but remain either architecture-specific or limited in optimization capacity to handle high-dimensional models. We propose ZOTTA, a fully BP-free TTA framework that performs efficient adaptation using only forward passes via Zeroth-Order Optimization (ZOO). While ZOO is theoretically appealing, naive application leads to slow convergence under high-dimensional parameter spaces and unstable optimization due to the lack of labels. ZOTTA overcomes these challenges through 1) Distribution-Robust Layer Selection, which automatically identifies and freezes layers that already extract distribution-invariant features, updating only domain-sensitive layers to reduce the optimization dimensionality and accelerate convergence; 2) Spatial Feature Aggregation Alignment, which stabilizes ZOO by aligning globally aggregated spatial features between source and target to reduce gradient variance. Together, these components enable architecture-agnostic and stable BP-free adaptation. Extensive experiments on ImageNet-C/R/Sketch/A show that ZOTTA outperforms or matches BP-based methods, e.g., it reduces memory usage by 84% and improves accuracy by 3.9% over SAR on ImageNet-C.
BibTeX
@misc{zhang2026zotta,
  author = {Ronghao Zhang and Shuaicheng Niu and Qi Deng and Yanjie Dong and Jian Chen and Runhao Zeng},
  title = {{ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimization}},
  year = {2026},
  eprint = {2603.14254},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.14254},
}
Figure