Enhancing Zeroth-Order Fine-Tuning for LLMs via Gradient-Guided Subspace Selection

Tongyifan Lin, Pengyu Zhang, Ming Hu, Xian Wei, Mingsong ChenSubmission to NeurIPS 2025 · Rejected

Abstract

As a promising memory-efficient technique, zeroth-order (ZO) optimization enables large language models (LLMs) to bypass costly backpropagation during fine-tuning by estimating gradients through function evaluations. However, to minimize approximate variance in high-dimensional parameter spaces, existing ZO methods focus on exploring the estimate of gradients within random subspaces, neglecting the benefits of searching for more accurate subspaces of LLMs on gradient estimates. Due to inaccurate gradient estimates obtained from random spaces, fine-tuning performance is inevitably degraded, thus compromising the performance of downstream tasks. To address the limitation of existing ZO methods, this paper proposes a novel ZO subspace fine-tuning method named *SVD-0*. Based on singular value decomposition (SVD), SVD-0 can effectively obtain more accurate subspace projection matrices, which can be used to improve the accuracy of gradient estimates. Experimental results on various complex language modeling tasks show that SVD-0 achieves better fine-tuning performance and faster convergence than state-of-the-art ZO methods.
BibTeX
@misc{lin2026enhancing,
  author = {Tongyifan Lin and Pengyu Zhang and Ming Hu and Xian Wei and Mingsong Chen},
  title = {{Enhancing Zeroth-Order Fine-Tuning for LLMs via Gradient-Guided Subspace Selection}},
  year = {2026},
  url = {https://openreview.net/forum?id=nlgQsugmGw},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}

Loading related content...

Figure