Backpropagation-Free Adaptation of Foundation Models

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2134 papers169 conceptsfrozen 2026-10-04

The primary barrier to fine-tuning Large Language Models (LLMs) on consumer hardware is not the size of the model weights, but the "memory wall" created by backpropagation. To compute a gradient, standard first-order optimizers must store the intermediate activations for every layer during the forward pass. For a multi-billion parameter model, these activations can consume tens of gigabytes of VRAM—far exceeding the capacity of most edge devices and even many high-end GPUs.

A rapidly growing body of research proposes a radical solution: Zeroth-Order (ZO) Optimization. By estimating gradients using only forward passes, these methods eliminate the need for activation caching, reducing the memory footprint of training to nearly the same level as simple inference.

The Intuition: Optimization as a Black Box

Standard training (first-order) is like having a topographic map of a mountain; you know exactly which direction is "down" because you have the gradient. Zeroth-order optimization is like being on that same mountain in total darkness with only an altimeter. To find the way down, you take a tiny step in a random direction, check if your altitude decreased, and then decide whether to move that way or the opposite way.

In the context of LLMs, this is implemented via SPSA (Simultaneous Perturbation Stochastic Approximation) or similar finite-difference methods. As described in Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark, the optimizer applies a small random perturbation vector to the model weights, performs a forward pass to measure the change in loss, and then updates the weights based on that difference. Because this requires only the output of the loss function, the model's internal mechanics remain a "black box."

Breaking the Memory Wall

The most significant advantage of this approach is hardware accessibility. By bypassing the storage of intermediate activations, researchers like Zhenqing Ling and Daoyuan Chen have demonstrated that billion-parameter models can be tuned on devices previously thought impossible.

  • On-Device Personalization: Zhihui Fu and Dan Peng demonstrated with PocketLLM (/work/pocketllm-enabling-on-device-fine-tuning-for-personalized) that models like OPT-1.3B can be fine-tuned directly on commercial smartphones.
  • Inference Engine Training: Frameworks like MobiZO, developed by Amir Ziashahabi and Salman Avestimehr, allow training to occur using standard inference engines that lack automatic differentiation capabilities entirely (/work/mobizo-enabling-efficient-llm-fine-tuning).
  • Quantization-Aware Training: Qitao Tan introduced ZeroQAT (/work/end-to-end-on-device-quantization-aware-training), which enables high-quality quantization-aware training for a 13B model on a single 8GB GPU.

Overcoming the Curse of Dimensionality

The traditional weakness of ZO optimization is that gradient estimates become noisier as the number of parameters increases. To make this viable for LLMs, recent work has introduced several stabilizing mechanisms:

  1. Subspace Optimization: Rather than perturbing every weight, methods like those proposed by Yuanhang Zheng in BSL (/work/black-box-prompt-tuning-with-subspace-learning) find a low-dimensional subspace where the most impactful updates reside.
  2. Block-Coordinate Updates: Qining Zhang and Lei Ying suggest perturbing only a subset of parameters at a time, which scales better to high-dimensional architectures (/work/zeroth-order-policy-gradient-for-reinforcement-learning).
  3. Hybrid Optimizers: Zeman Li and Meisam Razaviyayn developed Addax (/work/addax-utilizing-zeroth-order-gradients-to-improve-memory), which combines first-order and zeroth-order estimates to match the convergence speed of standard SGD while maintaining memory efficiency.

Privacy and Decentralization

Beyond memory savings, ZO optimization offers unique advantages for security and privacy. In Federated Learning, where models are trained across many decentralized devices, ZO methods like CYBER-0 (/work/communication-efficient-byzantine-resilient-federated-zero) reduce communication costs by orders of magnitude. Instead of sending massive gradient tensors, devices only need to communicate scalar loss values or small perturbation seeds.

Furthermore, Liang Zhang and Niao He have explored DPZero (/work/dpzero-dimension-independent-and-differentially-private), showing that zeroth-order methods can simplify Differentially Private (DP) training. Because ZO methods don't require per-example gradients, they bypass the memory-intensive "clipping and noising" steps that typically make DP training of large models prohibitively expensive.

The Future of Forward-Only Learning

The shift toward backpropagation-free adaptation represents a move away from specialized "training" infrastructure toward a unified "inference-as-training" paradigm. As models grow and edge deployment becomes mandatory, the ability to adapt a model using the same forward-pass logic used for generation—as seen in Zhen-Yan Lu’s MobiEdit (/work/mobiedit-resource-efficient-knowledge-editing)—may become the standard for personalized AI.

Papers in this version

Showing 60 of 134 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 97 papers
  2. Version 1 · 2026-10-04 · 134 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_zeroth_order_gradient_estimation_activat_7ccdf55c,
  title = {zeroth-order gradient estimation (activation-free optimization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zeroth-order-gradient-estimation-activation-free-optimization/rdr_7ccdf55cd1154773a1a213744baf0c40},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_backpropagation_free_adaptation_of_found_7ccdf55c_v1,
  title = {Backpropagation-Free Adaptation of Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/backpropagation-free-adaptation-of-foundation-models/rdr_7ccdf55cd1154773a1a213744baf0c40/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}