TL;DR
Group-wise preference optimization aligns Large Language Models (LLMs) more effectively by evaluating multiple candidate responses simultaneously, but it suffers from severe memory bottlenecks. This paper introduces a memory-efficient surrogate loss that mathematically matches the first-order gradients of group objectives while decoupling samples during backpropagation. This trick keeps memory usage flat as group sizes scale, unlocking stable, high-performance LLM alignment without running out of VRAM.
What Problem Does This Paper Address?
Standard Direct Preference Optimization (DPO) typically trains on pairs of data—one positive (chosen) and one negative (rejected) response per prompt. However, real-world preference datasets often contain multiple candidate responses per prompt. Binarizing this data into simple pairs throws away valuable context about the relative quality of the remaining responses.
While "group-wise" or "list-wise" optimization methods can leverage these extra responses for richer supervision, they introduce cross-response dependencies. Because the loss function evaluates interactions across the entire group, naive implementations must hold the activations for all responses in memory simultaneously during backpropagation. This quickly leads to Out-Of-Memory (OOM) errors, limiting researchers to very small group sizes.
Methods
The authors present a unified framework to solve this computational bottleneck using a memory-efficient surrogate implementation:
- First-Order Linearization: Instead of building a massive joint computation graph for the whole group, they run a lightweight, no-gradient forward pass to compute the implicit preference scores for all responses.
- Per-Sample Coefficients: Using these scores, they calculate objective-specific, per-response coefficients ($c_i$) that mathematically capture the cross-response interactions.
- Decoupled Backpropagation: They then optimize a sample-level surrogate loss ($\mathcal{L}_{sur} = \sum c_i u_i$), treating the precomputed coefficients as constants. This matches the exact first-order gradients of the original group objective but allows standard token-level backpropagation on individual samples.
Additionally, the authors integrate a negative log-likelihood (NLL) loss term on positive responses to stabilize the policy model during group-wise training.
Datasets
The approach was evaluated across both offline and online alignment settings:
- Offline Setting: Prompts sourced from
Dolci-Instruct-DPO. Candidate responses were generated using a pool of 16 diverse models (including Gemma, Olmo, Qwen, and Phi) and scored usingSkywork-Reward-V2-Qwen3-8Bto form positive and negative groups. - Online Setting: Prompts from
DAPO-Math-17k, where responses were sampled dynamically from the active policy model and partitioned based on rule-based correctness. - Evaluation Benchmarks: A massive suite covering math (AIME24/25, AMC23, MATH500, Minerva, OlympiadBench), reasoning (AGIEval, GPQA-D, MMLU-Pro, ZebraLogic, BBH), coding (HumanEval+, LiveCodeBench v6), and instruction following (IFBench, IFEval).
Key Insights and Ideas
- Rao-Blackwellization of DPO: For the All-Pairs objective, group-wise training acts as an explicit variance-reduction technique over standard random-pair DPO. Averaging over the group yields lower and more stable gradient norms during training.
- The Danger of Likelihood Collapse: Without the positive-response NLL regularization term, group-wise training is highly unstable. The model satisfies the contrastive objective by tanking the log probabilities of negative responses rather than raising the positives, leading to training collapse. Adding NLL completely mitigates this.
- Algorithmic Equivalence: Across different mathematical formulations of group-wise losses (Margin, MPO, All-Pairs, Softmax), the final downstream performance is remarkably similar. This implies that the primary benefit comes from the act of using multiple responses to shape prompt-level score separation, rather than the specific loss formula itself.
Key Results
- Consistent Performance Gains: Group-wise optimization consistently outperforms single-pair DPO and Rejection Fine-Tuning (RFT) across multiple model families (Gemma-3-4B, Olmo-3-7B, Olmo-3.1-32B, Qwen3-4B). For example, on Olmo-3-7B, group-wise training boosted the overall benchmark average from 48.5% (DPO) to over 51-52%.
- Group-Size Scalability: Performance steadily improves as group sizes scale from 2 to 8 responses, before plateauing at 16.
- Hardware Efficiency: The surrogate implementation successfully decouples memory consumption from group size. While the vanilla implementation crashes due to OOM errors at larger group sizes or longer sequence lengths, the surrogate maintains a flat, low memory footprint with only a modest latency overhead from the extra no-grad pass.
Limitations and Future Work
- First-Order Limitation: Because the surrogate loss relies on first-order linearization, it only preserves exact gradients for first-order optimizers (like AdamW or Muon). It discards second-order structure, meaning it might behave differently if paired with higher-order optimization algorithms.
- Compute Trade-off: The method requires an additional no-grad forward pass to precompute the coefficients, introducing a minor latency overhead compared to a hypothetical single-pass joint backpropagation (if memory were infinite).
Summary updated