TL;DR
Zeroth-order (ZO) optimization is a promising alternative to backpropagation for specialized hardware, but it usually fails in high-dimensional spaces due to the "curse of dimensionality." This paper introduces SZO-$\kappa$, a family of methods that use the geometry of activations from the forward pass to make "educated guesses" about gradient directions, significantly improving learning stability and performance on complex tasks like image classification.
What Problem Does This Paper Address?
Backpropagation is the gold standard for training deep networks, but it has a "hardware tax": it requires global synchrony, exact differentiability, and significant memory to store intermediate states. This makes it a poor fit for emerging "exotic" hardware like neuromorphic chips, asynchronous systems, or memory-constrained ASICs.
Zeroth-order (ZO) methods offer a "forward-only" alternative. They work by slightly perturbing the model, observing the change in the scalar loss, and correlating that change with the perturbation to estimate a gradient. However, in high-dimensional spaces, a random perturbation is almost certainly a "bad guess"—it’s likely to be nearly orthogonal to the true gradient. This leads to high-variance estimates that make training slow or unstable as networks get wider and deeper.
Methods
The authors propose moving beyond "blind" random noise. Their key insight is that the forward pass already contains information about which directions are worth exploring. Specifically, they look at the activation geometry: the directions in which the input data actually lives.
They derive a family of Spectral Zeroth-Order (SZO-$\kappa$) estimators. Instead of sampling isotropic (uniform) noise, SZO shapes the perturbation using the activation second-moment matrix $C = AA^\top$. The parameter $\kappa$ acts as a "bias-variance dial":
- $\kappa = 0$: Equivalent to standard, blind weight perturbation (WP).
- $\kappa = 1$: A "matched filter" that equalizes the signal-to-noise ratio across different feature directions.
- $\kappa \to \infty$: A rank-1 search that focuses entirely on the single most dominant feature direction (the leading eigenvector).
By choosing $\kappa > 0$, the algorithm prioritizes searching in directions that the current data actually exercises, ignoring "dead" directions that wouldn't change the output anyway.
Datasets
The researchers evaluated their methods using two standard benchmarks:
- Fashion-MNIST: Used to test a two-layer Multi-Layer Perceptron (MLP).
- CIFAR-10: Used to test a six-layer Convolutional Neural Network (CNN).
Key Insights and Ideas
- The Forward Pass is a Map: The authors argue that the forward pass isn't just for making predictions; it reveals the local "causal" landscape of the parameters. If a parameter direction doesn't move the activations for the current batch, it's a waste of a gradient probe.
- Unifying the Literature: SZO-$\kappa$ provides a theoretical framework that unifies existing methods. It shows that Node Perturbation (NP) and the recently proposed "hybrid" methods are actually specific points or limits within the SZO family.
- Solving the CNN Problem: A major "aha moment" is why Node Perturbation (NP) fails on convolutional layers. NP assumes nodes can be perturbed independently, but in a CNN, weights are shared across many spatial positions. SZO-1 and SZO-$\infty$ naturally respect this weight-sharing geometry, allowing them to outperform NP significantly on CIFAR-10.
Key Results
- Superior Alignment: SZO methods achieve gradient alignment (measured by the angle to the true backprop gradient) that far exceeds standard random perturbations, even with very few "probes."
- Training Performance: On CIFAR-10, SZO-1 reached significantly higher test accuracies than standard weight or node perturbation. While backprop still reaches the highest accuracy (~83.6%), SZO-1 narrows the gap considerably compared to previous zeroth-order attempts.
- Stability: The "educated" perturbations of SZO lead to much more stable training curves, especially when using adaptive optimizers like Adam or SOAP.
Limitations and Future Work
- Inter-layer Crosstalk: When all layers are perturbed simultaneously, noise from one layer can interfere with the gradient estimate of another. The authors used "iterative" perturbation (one layer at a time) to fix this, but this increases the number of forward passes needed.
- Isotropy Assumption: The derivation assumes that the "downstream" part of the network is equally sensitive to all changes. This is a good guess at the start of training but might become less accurate as the model matures.
- Computational Cost: While the authors found "decomposition-free" ways to implement SZO-1 and SZO-$\infty$, calculating the activation statistics still adds some overhead to the forward pass. Future work could focus on even more efficient ways to approximate these spectral directions.
Summary updated