Activation Free Fine Tuning via Zeroth Order Optimization

156 concepts90 papers1 proposal

When we train or fine-tune a deep learning model using standard first-order optimization algorithms like SGD or Adam, the primary memory bottleneck isn't the model's weights—it is the activations.

To compute the gradient of a loss function with respect to a weight in an intermediate layer via backpropagation, the chain rule requires the exact activation values computed during the forward pass. As a result, hardware must cache every intermediate activation tensor across all layers. This causes the training memory footprint to scale sharply with model depth, batch size, and sequence length, creating a "memory wall" that prevents large language models (LLMs) from being adapted on consumer-grade hardware or edge devices.

Activation-free optimization bypasses this bottleneck entirely by replacing backpropagation with zeroth-order (ZO) gradient estimation. By evaluating changes in the scalar loss function using only forward passes, this paradigm reduces the memory required for training to the exact same footprint as inference.


The Core Mechanism: Perturb and Evaluate

Zeroth-order optimization treats the entire neural network as a black box that maps parameters $\theta$ to a scalar loss $L(\theta)$. Instead of calculating analytical gradients via backpropagation, it estimates the gradient direction by applying small, random perturbations to the weights and observing how the loss responds.

To estimate the gradient vector for a model with millions or billions of parameters, the process follows three intuitive steps:

  1. Sample a Random Direction: We sample a random perturbation vector $z$ from a standard normal distribution. This vector has the exact same shape as the model parameters $\theta$.
  2. Evaluate the Loss Shift: We perform two forward passes (or one forward pass compared against an unperturbed baseline) to see how the scalar loss changes when we move slightly along the direction of $z$.
  3. Project Back to the Parameter Space: By observing the difference between the perturbed loss $L(\theta + \epsilon z)$ and the original loss $L(\theta)$, we can approximate the directional derivative of the loss along the vector $z$. We then scale $z$ by this scalar difference to form our gradient estimate $g$:

$$ g = \frac{L(\theta + \epsilon z) - L(\theta)}{\epsilon} z $$

As shown by Sadhika Malladi et al. in their work on Fine-Tuning Language Models with Just Forward Passes, this simple estimator (termed MeZO) allows full-parameter fine-tuning of massive language models using only forward passes. Because no backpropagation occurs, intermediate activations are discarded immediately after they are processed, dropping the activation memory cost to $O(1)$ with respect to the optimization step.

Standard Backpropagation (First-Order):
[Forward Pass: Compute & Cache Activations] ──> [Compute Loss] ──> [Backward Pass: Compute Gradients via Chain Rule]
Memory: High (Scales with layers, batch size, and sequence length)

Zeroth-Order Optimization (Activation-Free):
[Forward Pass 1: Base Loss L(θ)] ──> [Forward Pass 2: Perturbed Loss L(θ + εz)] ──> [Compute Scalar Difference & Update]
Memory: Low (Identical to inference; activations are discarded immediately)

The Memory Miracle: Seed-Based In-Place Management

A naive implementation of zeroth-order optimization would still hit a memory bottleneck: storing the random perturbation vector $z$ requires just as much memory as storing the model parameters themselves. For a 30-billion parameter model, keeping $z$ in memory would defeat the purpose of a low-overhead optimizer.

To circumvent this, the authors leverage a simple but powerful trick: random number generator (RNG) seeds.

Instead of storing the massive vector $z$ in memory, the optimizer only stores a single integer seed. During the forward pass, the perturbation is generated on-the-fly from the seed and added directly to the weights. Once the perturbed loss is computed, the perturbation is subtracted out to restore the original weights.

When it comes time to apply the parameter update, the exact same vector $z$ is reconstructed in-place by resetting the RNG to that identical seed. As highlighted by Zhenqing Ling et al. in On the Convergence of Zeroth-Order Federated Tuning for Large Language Models, this in-place management strategy ensures that the memory footprint remains virtually constant regardless of the model size, enabling memory-constrained edge devices to execute full-parameter updates.


Taming the Curse of Dimensionality

While activation-free optimization solves the memory bottleneck, it introduces a significant mathematical challenge: high variance.

Because the gradient is estimated along a single random direction $z$, the variance of the estimator scales linearly with the number of parameters $d$. In a model with billions of parameters, a naive ZO update can be incredibly noisy, leading to slow convergence or optimization instability.

To bridge the performance gap between zeroth-order and traditional first-order methods, recent research focuses on variance reduction, hybrid architectures, and subspace projections:


Shifting Training to the Edge

Because zeroth-order optimization reduces training memory to inference levels, it naturally pairs with quantization techniques designed to compress models for edge deployment.

For instance, Qitao Tan et al. introduced ZeroQAT in End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost, demonstrating that quantization-aware training can be executed directly on commercial smartphones by replacing backpropagation with forward-only perturbations. By eliminating both the activation memory wall and full-precision gradient storage, this paradigm shifts machine learning optimization away from massive data centers and directly onto resource-constrained edge hardware.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 97 papers
  2. Version 1 · 2026-10-04 · 134 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_zeroth_order_gradient_estimation_activat_7ccdf55c,
  title = {zeroth-order gradient estimation (activation-free optimization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zeroth-order-gradient-estimation-activation-free-optimization/rdr_7ccdf55cd1154773a1a213744baf0c40},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_zeroth_order_gradient_estimation_activat_7ccdf55c_v2,
  title = {zeroth-order gradient estimation (activation-free optimization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zeroth-order-gradient-estimation-activation-free-optimization/rdr_7ccdf55cd1154773a1a213744baf0c40/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake