An Educated Guess: Deriving Statistically Aligned Gradient Estimates for Zeroth-Order Optimisation

Justus Huebotter, Serge Thill, Marcel van Gerven, Nasir AhmadarXiv 2026 · arXiv

TL;DR

Zeroth-order (ZO) optimization is a promising alternative to backpropagation for specialized hardware, but it usually fails in high-dimensional spaces due to the "curse of dimensionality." This paper introduces SZO-$\kappa$, a family of methods that use the geometry of activations from the forward pass to make "educated guesses" about gradient directions, significantly improving learning stability and performance on complex tasks like image classification.

What Problem Does This Paper Address?

Backpropagation is the gold standard for training deep networks, but it has a "hardware tax": it requires global synchrony, exact differentiability, and significant memory to store intermediate states. This makes it a poor fit for emerging "exotic" hardware like neuromorphic chips, asynchronous systems, or memory-constrained ASICs.

Zeroth-order (ZO) methods offer a "forward-only" alternative. They work by slightly perturbing the model, observing the change in the scalar loss, and correlating that change with the perturbation to estimate a gradient. However, in high-dimensional spaces, a random perturbation is almost certainly a "bad guess"—it’s likely to be nearly orthogonal to the true gradient. This leads to high-variance estimates that make training slow or unstable as networks get wider and deeper.

Methods

The authors propose moving beyond "blind" random noise. Their key insight is that the forward pass already contains information about which directions are worth exploring. Specifically, they look at the activation geometry: the directions in which the input data actually lives.

They derive a family of Spectral Zeroth-Order (SZO-$\kappa$) estimators. Instead of sampling isotropic (uniform) noise, SZO shapes the perturbation using the activation second-moment matrix $C = AA^\top$. The parameter $\kappa$ acts as a "bias-variance dial":

  • $\kappa = 0$: Equivalent to standard, blind weight perturbation (WP).
  • $\kappa = 1$: A "matched filter" that equalizes the signal-to-noise ratio across different feature directions.
  • $\kappa \to \infty$: A rank-1 search that focuses entirely on the single most dominant feature direction (the leading eigenvector).

By choosing $\kappa > 0$, the algorithm prioritizes searching in directions that the current data actually exercises, ignoring "dead" directions that wouldn't change the output anyway.

Datasets

The researchers evaluated their methods using two standard benchmarks:

  1. Fashion-MNIST: Used to test a two-layer Multi-Layer Perceptron (MLP).
  2. CIFAR-10: Used to test a six-layer Convolutional Neural Network (CNN).

Key Insights and Ideas

  • The Forward Pass is a Map: The authors argue that the forward pass isn't just for making predictions; it reveals the local "causal" landscape of the parameters. If a parameter direction doesn't move the activations for the current batch, it's a waste of a gradient probe.
  • Unifying the Literature: SZO-$\kappa$ provides a theoretical framework that unifies existing methods. It shows that Node Perturbation (NP) and the recently proposed "hybrid" methods are actually specific points or limits within the SZO family.
  • Solving the CNN Problem: A major "aha moment" is why Node Perturbation (NP) fails on convolutional layers. NP assumes nodes can be perturbed independently, but in a CNN, weights are shared across many spatial positions. SZO-1 and SZO-$\infty$ naturally respect this weight-sharing geometry, allowing them to outperform NP significantly on CIFAR-10.
Figure 1: The angles between estimated gradients computed by NP, WP, SZO-κ and NTWP against the true backpropagation gradient for randomly initialised layers in MLP and Convolu- tional Networks. For each network, the left panel uses per-batch loss-difference credit and the right panel per-sample credit; points show the mean angle over 20 randomly seeded initialisations (batch of 128) against the number of probes n, with error bars of ±1 standard deviation. Left pair, the gradient alignment angle is shown for the first layer of our Fashion-MNIST MLP network. Under per-sample credit these angles clearly show the equivalence of NP and SZO-κ on a linear layer, as theorised in this work. Right pair, the gradient alignment angle is shown for the last 3 × 3 convolutional layer (64 →64 channels) of our CIFAR-10 CNN model. On convolutional layers NP suffers significantly from its lack of shared perturbation across kernel positions. Note, these angles were measured by perturbing only the layer in question and leaving all other network layers unperturbed.

Key Results

  • Superior Alignment: SZO methods achieve gradient alignment (measured by the angle to the true backprop gradient) that far exceeds standard random perturbations, even with very few "probes."
  • Training Performance: On CIFAR-10, SZO-1 reached significantly higher test accuracies than standard weight or node perturbation. While backprop still reaches the highest accuracy (~83.6%), SZO-1 narrows the gap considerably compared to previous zeroth-order attempts.
  • Stability: The "educated" perturbations of SZO lead to much more stable training curves, especially when using adaptive optimizers like Adam or SOAP.
Figure 2: The performance of the SZO-family of methods as well as NP, WP, and NTWP when training an MLP (Fashion-MNIST) and a Convolutional Network (CIFAR-10). (a,c) Test accuracy over training with SGD and n = 1 (mean over 10 seeds, ±1 standard deviation); on the MLP, NP, SZO-1 and SZO-∞coincide and are drawn interleaved; BP (not shown) reaches 89.9% and 83.6%. (b,d) Validation accuracy after 250 epochs against the number of probes per step n (mean over 3 seeds, 95% confidence intervals; Appendix M). The learning rate of each method was selected via a grid search on validation loss. Final test accuracies for each method, as well as backpropagation, are reported in Appendix L.

Limitations and Future Work

  • Inter-layer Crosstalk: When all layers are perturbed simultaneously, noise from one layer can interfere with the gradient estimate of another. The authors used "iterative" perturbation (one layer at a time) to fix this, but this increases the number of forward passes needed.
  • Isotropy Assumption: The derivation assumes that the "downstream" part of the network is equally sensitive to all changes. This is a good guess at the start of training but might become less accurate as the model matures.
  • Computational Cost: While the authors found "decomposition-free" ways to implement SZO-1 and SZO-$\infty$, calculating the activation statistics still adds some overhead to the forward pass. Future work could focus on even more efficient ways to approximate these spectral directions.

Summary updated

Abstract

As novel hardware architectures are explored for the purposes of machine learning, including neuromorphic chips, asynchronous systems, and ASICs, effective learning algorithms are highly desirable. While backpropagation is the standard for differentiation on standard hardware, its requirement for global synchrony and exact differentiability limits the exploration of exotic algorithms on non-traditional substrates and devices. Zeroth-order (ZO) optimisation methods, such as SPSA and weight perturbation, offer a compelling alternative by estimating gradients through inference-only passes; however, these methods have historically suffered due to the `curse of dimensionality', where random perturbations become increasingly misaligned with the true gradient in high-dimensional spaces. In this work, we move beyond blind random noise by deriving zeroth-order search directions from activation geometry available during the forward pass. This leads to a family of Spectral Zeroth-Order methods, SZO-$κ$, whose directions solve local response-based gradient-guessing objectives under explicit stochastic assumptions. These methods explore feature-supported directions that are more likely to produce informative loss responses. Using multilayer and convolutional networks, we analyse gradient alignment, bias-variance structure, and training stability of these methods relative to standard perturbation baselines and backpropagation.
BibTeX
@misc{huebotter2026educated,
  author = {Justus Huebotter and Serge Thill and Marcel van Gerven and Nasir Ahmad},
  title = {{An Educated Guess: Deriving Statistically Aligned Gradient Estimates for Zeroth-Order Optimisation}},
  year = {2026},
  eprint = {2611.08712},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08712},
}
Figure