When we train or fine-tune a deep learning model using standard first-order optimization algorithms like SGD or Adam, the primary memory bottleneck isn't the model's weights—it is the activations.
To compute the gradient of a loss function with respect to a weight in an intermediate layer via backpropagation, the chain rule requires the exact activation values computed during the forward pass. As a result, hardware must cache every intermediate activation tensor across all layers. This causes the training memory footprint to scale sharply with model depth, batch size, and sequence length, creating a "memory wall" that prevents large language models (LLMs) from being adapted on consumer-grade hardware or edge devices.
Activation-free optimization bypasses this bottleneck entirely by replacing backpropagation with zeroth-order (ZO) gradient estimation. By evaluating changes in the scalar loss function using only forward passes, this paradigm reduces the memory required for training to the exact same footprint as inference.
The Core Mechanism: Perturb and Evaluate
Zeroth-order optimization treats the entire neural network as a black box that maps parameters $\theta$ to a scalar loss $L(\theta)$. Instead of calculating analytical gradients via backpropagation, it estimates the gradient direction by applying small, random perturbations to the weights and observing how the loss responds.
To estimate the gradient vector for a model with millions or billions of parameters, the process follows three intuitive steps:
- Sample a Random Direction: We sample a random perturbation vector $z$ from a standard normal distribution. This vector has the exact same shape as the model parameters $\theta$.
- Evaluate the Loss Shift: We perform two forward passes (or one forward pass compared against an unperturbed baseline) to see how the scalar loss changes when we move slightly along the direction of $z$.
- Project Back to the Parameter Space: By observing the difference between the perturbed loss $L(\theta + \epsilon z)$ and the original loss $L(\theta)$, we can approximate the directional derivative of the loss along the vector $z$. We then scale $z$ by this scalar difference to form our gradient estimate $g$:
$$ g = \frac{L(\theta + \epsilon z) - L(\theta)}{\epsilon} z $$
As shown by Sadhika Malladi et al. in their work on Fine-Tuning Language Models with Just Forward Passes, this simple estimator (termed MeZO) allows full-parameter fine-tuning of massive language models using only forward passes. Because no backpropagation occurs, intermediate activations are discarded immediately after they are processed, dropping the activation memory cost to $O(1)$ with respect to the optimization step.
Standard Backpropagation (First-Order):
[Forward Pass: Compute & Cache Activations] ──> [Compute Loss] ──> [Backward Pass: Compute Gradients via Chain Rule]
Memory: High (Scales with layers, batch size, and sequence length)
Zeroth-Order Optimization (Activation-Free):
[Forward Pass 1: Base Loss L(θ)] ──> [Forward Pass 2: Perturbed Loss L(θ + εz)] ──> [Compute Scalar Difference & Update]
Memory: Low (Identical to inference; activations are discarded immediately)
The Memory Miracle: Seed-Based In-Place Management
A naive implementation of zeroth-order optimization would still hit a memory bottleneck: storing the random perturbation vector $z$ requires just as much memory as storing the model parameters themselves. For a 30-billion parameter model, keeping $z$ in memory would defeat the purpose of a low-overhead optimizer.
To circumvent this, the authors leverage a simple but powerful trick: random number generator (RNG) seeds.
Instead of storing the massive vector $z$ in memory, the optimizer only stores a single integer seed. During the forward pass, the perturbation is generated on-the-fly from the seed and added directly to the weights. Once the perturbed loss is computed, the perturbation is subtracted out to restore the original weights.
When it comes time to apply the parameter update, the exact same vector $z$ is reconstructed in-place by resetting the RNG to that identical seed. As highlighted by Zhenqing Ling et al. in On the Convergence of Zeroth-Order Federated Tuning for Large Language Models, this in-place management strategy ensures that the memory footprint remains virtually constant regardless of the model size, enabling memory-constrained edge devices to execute full-parameter updates.
Taming the Curse of Dimensionality
While activation-free optimization solves the memory bottleneck, it introduces a significant mathematical challenge: high variance.
Because the gradient is estimated along a single random direction $z$, the variance of the estimator scales linearly with the number of parameters $d$. In a model with billions of parameters, a naive ZO update can be incredibly noisy, leading to slow convergence or optimization instability.
To bridge the performance gap between zeroth-order and traditional first-order methods, recent research focuses on variance reduction, hybrid architectures, and subspace projections:
- Variance Reduction: To stabilize the optimization trajectory, Tanmay Gautam et al. developed MeZO-SVRG in Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models. This approach applies stochastic variance-reduced gradient techniques to ZO optimization, cutting down gradient noise and accelerating convergence.
- Hybrid Optimization: Instead of using ZO for the entire network, the authors of New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework proposed a mixed-order strategy. They update the massive base model parameters using memory-efficient ZO optimization, while simultaneously training small parameter-efficient modules (like LoRA adapters) using standard first-order backpropagation. This allows the network to capture fine-grained updates without increasing peak memory usage.
- Subspace Projection and Bias Reduction: Rather than searching the entire high-dimensional parameter space, Hoigi Seo et al. introduced a method in Efficient Personalization of Quantized Diffusion Model without Backpropagation that denoises gradient estimates by projecting them into a lower-dimensional subspace. To further improve accuracy, Zhendong Mi et al. developed KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning, which utilizes kernel functions to systematically mitigate gradient estimation bias.
Shifting Training to the Edge
Because zeroth-order optimization reduces training memory to inference levels, it naturally pairs with quantization techniques designed to compress models for edge deployment.
For instance, Qitao Tan et al. introduced ZeroQAT in End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost, demonstrating that quantization-aware training can be executed directly on commercial smartphones by replacing backpropagation with forward-only perturbations. By eliminating both the activation memory wall and full-precision gradient storage, this paradigm shifts machine learning optimization away from massive data centers and directly onto resource-constrained edge hardware.