OAT-Rephrase: Optimization-Aware Training Data Rephrasing for Zeroth-Order LLM Fine-Tuning

Jikai Long, Zijian Hu, Xiaodong Yu, Jianwen Xie, Zhaozhuo XuarXiv 2025 · DOI · arXiv

Abstract

Zeroth-Order (ZO) optimization methods like MeZO offer a memory-efficient alternative to traditional First-Order (FO) fine-tuning for Large Language Models (LLMs), enabling training on consumer-grade hardware. However, ZO methods often suffer from high gradient variance and slow convergence. This paper introduces OAT-Rephrase (Optimization-Aware Training Data Rephrasing), a data-centric approach that uses an LLM-based rewriter to transform training data into a more "optimization-friendly" format. By smoothing the loss landscape, OAT-Rephrase significantly enhances the stability and accuracy of ZO fine-tuning. Across five benchmarks, OAT-Rephrase achieves an average accuracy improvement of over 2%, occasionally outperforming standard FO full fine-tuning.

Introduction

Fine-tuning Large Language Models (LLMs) typically relies on First-Order (FO) optimization via backpropagation. While effective, the memory overhead of storing intermediate activations limits its feasibility on hardware with restricted VRAM. Zeroth-Order (ZO) optimization, specifically the MeZO algorithm, addresses this by estimating gradients using only forward passes. Despite its memory efficiency, ZO is plagued by high variance in gradient estimates, leading to unstable training and slow convergence.

OAT-Rephrase posits that the problem is not solely algorithmic but also data-centric. Noisy or ambiguous phrasing in training data creates a "jagged" loss landscape that makes ZO's rough gradient estimates even less reliable. By rephrasing data to be more "semantically stable," the authors aim to create a smoother loss surface, making finite-difference gradient estimates much more accurate.

Methodology

The OAT-Rephrase framework consists of a dual-stage pipeline: an LLM-based Rewriter and a Semantic Judge.

The Rewriter

The Rewriter utilizes GPT-4o, which is prompted with the technical details of the MeZO research paper. Through a sophisticated five-phase prompt engineering process, the rewriter is instructed to modify training instances to be clearer and more stable for the MeZO algorithm while maintaining the original task's logic. This "Optimization-Awareness" ensures the rewriter isn't just paraphrasing for style, but specifically trying to create inputs that yield more stable loss values.

The Semantic Judge

To ensure data integrity and prevent "hallucinations" or label leakage, a second LLM instance acts as a Semantic Judge. It compares the original and rephrased versions; if the meaning has drifted or the logic is broken, the rewrite is rejected, and the original data is kept.

The OAT-Rephrase pipeline: The rewriter generates candidates which are then verified by a semantic judge to ensure task consistency.

Experimental Results

The researchers evaluated OAT-Rephrase on five diverse English classification benchmarks: COPA (causal reasoning), CB (linguistic entailment), RTE (textual entailment), BoolQ (reading comprehension), and ARC-Challenge (science QA). These were tested across multiple architectures, including Llama-3.2 (1B and 3B) and Mistral-7B.

OAT-Rephrase consistently improved MeZO performance across nearly all tasks and models. For Llama-1B, the average accuracy increased by 2.28%. Notably, in several instances, MeZO combined with OAT-Rephrase actually outperformed traditional First-Order full fine-tuning. For example, on the CB dataset with Llama-1B, the OAT-MeZO approach reached 71.4% accuracy, while standard full fine-tuning only hit 66.1%.

Performance comparison of ZO-MeZO fine-tuning on original vs. OAT-Rephrased data across five benchmarks.

Discussion and Conclusion

A key insight of this work is that "ZO-friendliness" appears to be a general property of how a task is presented. Data rephrased by a powerful model like GPT-4o improved the training of other models like Llama and Mistral, suggesting cross-model generalization. By rephrasing data to be more semantically stable, OAT-Rephrase creates a smoother loss surface, which is beneficial regardless of the specific model weights.

While the method introduces a one-time overhead for rephrasing, the subsequent fine-tuning remains as memory-efficient as standard ZO. Future work will focus on extending this approach to generative tasks and automating safety checks to prevent label leakage more robustly. Overall, OAT-Rephrase demonstrates that data-centric interventions can effectively bridge the performance gap between memory-efficient Zeroth-Order methods and traditional gradient-based training.

Summary updated

Abstract

Fine-tuning large language models (LLMs) using zeroth-order optimization (ZO) offers a memory-efficient alternative to gradient-based methods but suffers from slower convergence and unstable optimization due to noisy gradient estimates. This paper introduces OAT-Rephrase, an Optimization-Aware Training data rephrasing strategy that leverages an LLM to rephrase training instances based on its understanding of the ZO dynamics, specifically MeZO, derived directly from its paper. The approach incorporates a dual-stage pipeline featuring a rewriter LLM and a semantic judge, ensuring all rephrasings retain task relevance and logical consistency. Evaluations across five classification tasks and three LLM architectures demonstrate that OAT-Rephrase consistently improves MeZO fine-tuning performance, often narrowing or eliminating the gap with first-order methods. Our findings suggest that optimization-aware rephrasing serves as a reusable and low-overhead enhancement for zeroth-order tuning regimes.
BibTeX
@misc{long2025oat,
  author = {Jikai Long and Zijian Hu and Xiaodong Yu and Jianwen Xie and Zhaozhuo Xu},
  title = {{OAT-Rephrase: Optimization-Aware Training Data Rephrasing for Zeroth-Order LLM Fine-Tuning}},
  year = {2025},
  eprint = {2506.17264},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.17264},
}
Figure