TL;DR
CA-ZODPO is a new memory-efficient method for aligning Large Language Models (LLMs) that slashes peak GPU memory usage by nearly 60% compared to standard Direct Preference Optimisation (DPO). By restricting parameter updates to a low-rank subspace guided by the difference in activations between preferred and rejected responses, it achieves performance competitive with traditional backpropagation-based methods while avoiding their massive memory overhead.
What Problem Does This Paper Address?
Aligning LLMs with human preferences typically involves Direct Preference Optimisation (DPO), which relies on backpropagation. The catch? Backpropagation requires storing intermediate activations for every layer, leading to a "memory wall." During training, memory costs can be up to 12 times higher than during inference, making it difficult to align large models or process long sequences on standard hardware.
Zeroth-Order (ZO) optimization offers a way out by estimating gradients using only forward passes (no activation caching required). However, existing ZO methods like MeZO are often too "noisy." They perturb parameters in random directions, which is inefficient for preference learning where the signal depends on the subtle distinction between a "good" (chosen) and "bad" (rejected) response. The authors aim to bridge the performance gap between memory-efficient ZO methods and high-performing first-order methods.
Methods
The authors introduce Contrastive Activation-guided Zeroth-Order Direct Preference Optimisation (CA-ZODPO). The core innovation lies in how the algorithm chooses the directions to "wiggle" the model's parameters:
- Contrastive Subspace Construction: For each linear layer, the model performs a forward pass on a preference pair. It calculates the difference between the activations of the chosen response and the rejected response.
- Low-Rank Projection: Using power iteration, the algorithm extracts a low-rank basis (a small set of important directions) from this activation discrepancy.
- Guided Perturbation: Instead of perturbing parameters randomly across the entire high-dimensional space, CA-ZODPO only applies perturbations within this specific, task-relevant subspace.
- Low-Rank Momentum: To further save memory, the optimizer stores momentum only for these low-rank coordinates rather than for every single parameter in the model.
- Two-Sided Estimation: It uses a central finite-difference estimator (evaluating the loss at $+\epsilon$ and $-\epsilon$ perturbations) to get a more stable gradient estimate.
Datasets
The researchers evaluated their method using:
- Model: Qwen2.5-1.5B as the base model.
- Training Data: The Anthropic HH-RLHF (Helpful and Harmless) dataset, a standard benchmark for preference alignment.
- Evaluation: Pairwise comparisons judged by a larger model, Qwen2.5-32B-Instruct, to determine which training method produced better responses.
Key Insights and Ideas
- The Power of "Difference": A key finding was that simply using activations (as in prior work like AGZO) isn't enough for DPO. The contrastive element—specifically looking at the discrepancy between chosen and rejected activations—is what allows the optimizer to find the most meaningful directions for alignment.
- Breaking the Sequence-Length Barrier: In standard DPO, memory usage spikes as the sequence length increases because more activations must be stored. CA-ZODPO retains the "flat" memory scaling of zeroth-order methods, making it a prime candidate for long-context alignment tasks.
- Subspace Stability: The authors found that a very low rank (e.g., $r=1$) can be unstable, but increasing the rank slightly (to $r=4$ or $r=8$) significantly stabilizes the training trajectory, narrowing the gap to first-order DPO.
Key Results
- Memory Savings: At a sequence length of 2048, CA-ZODPO used only 11.08 GB of peak GPU memory, compared to 26.94 GB for standard first-order DPO—a 59% reduction.
- Superiority over MeZO: While the standard ZO method (MeZO) actually performed worse than the starting SFT (Supervised Fine-Tuning) policy, CA-ZODPO showed clear improvement, achieving a 34.6% win rate against SFT.
- Competitive Performance: CA-ZODPO proved nearly as effective as standard DPO. In head-to-head pairwise evaluations, CA-ZODPO achieved a 34.2% win rate against first-order DPO, resulting in a nearly balanced win/loss profile.
Limitations and Future Work
- Computational Overhead: While CA-ZODPO saves memory, it requires three forward passes per update (one for subspace extraction and two for the gradient estimate). This means it is memory-efficient but not necessarily faster than backpropagation.
- Scale and Diversity: The study focused on a 1.5B parameter model and a single dataset. The authors note that future work should explore how this scales to much larger models (e.g., 70B+) and more diverse preference datasets.
- LLM Judges: The evaluation relies on an LLM judge, which, while standard, can have its own biases. Further human evaluation could provide additional validation.
Summary updated