Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment

Weiyun Huang, Alessio GalatoloarXiv 2026 · arXiv

TL;DR

CA-ZODPO is a new memory-efficient method for aligning Large Language Models (LLMs) that slashes peak GPU memory usage by nearly 60% compared to standard Direct Preference Optimisation (DPO). By restricting parameter updates to a low-rank subspace guided by the difference in activations between preferred and rejected responses, it achieves performance competitive with traditional backpropagation-based methods while avoiding their massive memory overhead.

What Problem Does This Paper Address?

Aligning LLMs with human preferences typically involves Direct Preference Optimisation (DPO), which relies on backpropagation. The catch? Backpropagation requires storing intermediate activations for every layer, leading to a "memory wall." During training, memory costs can be up to 12 times higher than during inference, making it difficult to align large models or process long sequences on standard hardware.

Zeroth-Order (ZO) optimization offers a way out by estimating gradients using only forward passes (no activation caching required). However, existing ZO methods like MeZO are often too "noisy." They perturb parameters in random directions, which is inefficient for preference learning where the signal depends on the subtle distinction between a "good" (chosen) and "bad" (rejected) response. The authors aim to bridge the performance gap between memory-efficient ZO methods and high-performing first-order methods.

Methods

The authors introduce Contrastive Activation-guided Zeroth-Order Direct Preference Optimisation (CA-ZODPO). The core innovation lies in how the algorithm chooses the directions to "wiggle" the model's parameters:

  1. Contrastive Subspace Construction: For each linear layer, the model performs a forward pass on a preference pair. It calculates the difference between the activations of the chosen response and the rejected response.
  2. Low-Rank Projection: Using power iteration, the algorithm extracts a low-rank basis (a small set of important directions) from this activation discrepancy.
  3. Guided Perturbation: Instead of perturbing parameters randomly across the entire high-dimensional space, CA-ZODPO only applies perturbations within this specific, task-relevant subspace.
  4. Low-Rank Momentum: To further save memory, the optimizer stores momentum only for these low-rank coordinates rather than for every single parameter in the model.
  5. Two-Sided Estimation: It uses a central finite-difference estimator (evaluating the loss at $+\epsilon$ and $-\epsilon$ perturbations) to get a more stable gradient estimate.

Datasets

The researchers evaluated their method using:

  • Model: Qwen2.5-1.5B as the base model.
  • Training Data: The Anthropic HH-RLHF (Helpful and Harmless) dataset, a standard benchmark for preference alignment.
  • Evaluation: Pairwise comparisons judged by a larger model, Qwen2.5-32B-Instruct, to determine which training method produced better responses.

Key Insights and Ideas

  • The Power of "Difference": A key finding was that simply using activations (as in prior work like AGZO) isn't enough for DPO. The contrastive element—specifically looking at the discrepancy between chosen and rejected activations—is what allows the optimizer to find the most meaningful directions for alignment.
  • Breaking the Sequence-Length Barrier: In standard DPO, memory usage spikes as the sequence length increases because more activations must be stored. CA-ZODPO retains the "flat" memory scaling of zeroth-order methods, making it a prime candidate for long-context alignment tasks.
  • Subspace Stability: The authors found that a very low rank (e.g., $r=1$) can be unstable, but increasing the rank slightly (to $r=4$ or $r=8$) significantly stabilizes the training trajectory, narrowing the gap to first-order DPO.

Key Results

  • Memory Savings: At a sequence length of 2048, CA-ZODPO used only 11.08 GB of peak GPU memory, compared to 26.94 GB for standard first-order DPO—a 59% reduction.
  • Superiority over MeZO: While the standard ZO method (MeZO) actually performed worse than the starting SFT (Supervised Fine-Tuning) policy, CA-ZODPO showed clear improvement, achieving a 34.6% win rate against SFT.
  • Competitive Performance: CA-ZODPO proved nearly as effective as standard DPO. In head-to-head pairwise evaluations, CA-ZODPO achieved a 34.2% win rate against first-order DPO, resulting in a nearly balanced win/loss profile.

Limitations and Future Work

  • Computational Overhead: While CA-ZODPO saves memory, it requires three forward passes per update (one for subspace extraction and two for the gradient estimate). This means it is memory-efficient but not necessarily faster than backpropagation.
  • Scale and Diversity: The study focused on a 1.5B parameter model and a single dataset. The authors note that future work should explore how this scales to much larger models (e.g., 70B+) and more diverse preference datasets.
  • LLM Judges: The evaluation relies on an LLM judge, which, while standard, can have its own biases. Further human evaluation could provide additional validation.

Summary updated

Abstract

Direct Preference Optimisation (DPO) simplifies reinforcement learning from human feedback, but still relies on traditional backpropagation and its associated activation-memory cost. Zeroth-order (ZO) optimisation avoids backpropagation, yet isotropic perturbations can produce noisy updates for preference objectives defined by differences between chosen and rejected responses. We introduce Contrastive Activation-guided Zeroth-Order Direct Preference Optimisation (CA-ZODPO), which constructs a low-rank search subspace from the activation discrepancy between the chosen and rejected responses in each preference pair. Perturbations are restricted to this subspace, while momentum is maintained in low-rank coordinates to preserve the memory advantages of ZO training. On Qwen2.5-1.5B with Anthropic HH-RLHF, CA-ZODPO outperforms common ZO baselines and remains competitive with first-order DPO in pairwise preference evaluation. It also reduces peak GPU memory by about 59\% at a sequence length of 2048 while retaining the sequence-length scaling behaviour of zeroth-order training.
BibTeX
@misc{huang2026contrastive,
  author = {Weiyun Huang and Alessio Galatolo},
  title = {{Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment}},
  year = {2026},
  eprint = {2611.20635},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20635},
}
Figure