Abstract
Zeroth-Order (ZO) optimization methods like MeZO offer a memory-efficient alternative to traditional First-Order (FO) fine-tuning for Large Language Models (LLMs), enabling training on consumer-grade hardware. However, ZO methods often suffer from high gradient variance and slow convergence. This paper introduces OAT-Rephrase (Optimization-Aware Training Data Rephrasing), a data-centric approach that uses an LLM-based rewriter to transform training data into a more "optimization-friendly" format. By smoothing the loss landscape, OAT-Rephrase significantly enhances the stability and accuracy of ZO fine-tuning. Across five benchmarks, OAT-Rephrase achieves an average accuracy improvement of over 2%, occasionally outperforming standard FO full fine-tuning.
Introduction
Fine-tuning Large Language Models (LLMs) typically relies on First-Order (FO) optimization via backpropagation. While effective, the memory overhead of storing intermediate activations limits its feasibility on hardware with restricted VRAM. Zeroth-Order (ZO) optimization, specifically the MeZO algorithm, addresses this by estimating gradients using only forward passes. Despite its memory efficiency, ZO is plagued by high variance in gradient estimates, leading to unstable training and slow convergence.
OAT-Rephrase posits that the problem is not solely algorithmic but also data-centric. Noisy or ambiguous phrasing in training data creates a "jagged" loss landscape that makes ZO's rough gradient estimates even less reliable. By rephrasing data to be more "semantically stable," the authors aim to create a smoother loss surface, making finite-difference gradient estimates much more accurate.
Methodology
The OAT-Rephrase framework consists of a dual-stage pipeline: an LLM-based Rewriter and a Semantic Judge.
The Rewriter
The Rewriter utilizes GPT-4o, which is prompted with the technical details of the MeZO research paper. Through a sophisticated five-phase prompt engineering process, the rewriter is instructed to modify training instances to be clearer and more stable for the MeZO algorithm while maintaining the original task's logic. This "Optimization-Awareness" ensures the rewriter isn't just paraphrasing for style, but specifically trying to create inputs that yield more stable loss values.
The Semantic Judge
To ensure data integrity and prevent "hallucinations" or label leakage, a second LLM instance acts as a Semantic Judge. It compares the original and rephrased versions; if the meaning has drifted or the logic is broken, the rewrite is rejected, and the original data is kept.
Experimental Results
The researchers evaluated OAT-Rephrase on five diverse English classification benchmarks: COPA (causal reasoning), CB (linguistic entailment), RTE (textual entailment), BoolQ (reading comprehension), and ARC-Challenge (science QA). These were tested across multiple architectures, including Llama-3.2 (1B and 3B) and Mistral-7B.
OAT-Rephrase consistently improved MeZO performance across nearly all tasks and models. For Llama-1B, the average accuracy increased by 2.28%. Notably, in several instances, MeZO combined with OAT-Rephrase actually outperformed traditional First-Order full fine-tuning. For example, on the CB dataset with Llama-1B, the OAT-MeZO approach reached 71.4% accuracy, while standard full fine-tuning only hit 66.1%.
Discussion and Conclusion
A key insight of this work is that "ZO-friendliness" appears to be a general property of how a task is presented. Data rephrased by a powerful model like GPT-4o improved the training of other models like Llama and Mistral, suggesting cross-model generalization. By rephrasing data to be more semantically stable, OAT-Rephrase creates a smoother loss surface, which is beneficial regardless of the specific model weights.
While the method introduces a one-time overhead for rephrasing, the subsequent fine-tuning remains as memory-efficient as standard ZO. Future work will focus on extending this approach to generative tasks and automating safety checks to prevent label leakage more robustly. Overall, OAT-Rephrase demonstrates that data-centric interventions can effectively bridge the performance gap between memory-efficient Zeroth-Order methods and traditional gradient-based training.
Summary updated