Weak-to-Strong Extrapolation Expedites Alignment

Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, Nanyun Peng

TL;DR

Aligning large language models with human preference via RLHF or DPO is computationally expensive and often results in undertrained alignment due to regularization constraints. ExPO (Weak-to-Strong Extrapolation) bypasses further training by simply extrapolating model weights along the direction from the initial SFT checkpoint to the aligned checkpoint ($\theta_2 = \theta_1 + \alpha(\theta_1 - \theta_0)$). This zero-training trick consistently boosts instruction-following and preference alignment across 12 open models ranging from 1.8B to 70B parameters.

What Problem Does This Paper Address?

Post-training large language models (LLMs) to match human preferences via reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO) requires substantial compute and preference annotations. Even after alignment, most open-source models remain suboptimally aligned. This under-optimization happens partly because of strict regularization terms (like the KL-divergence penalty that prevents models from drifting too far from the reference model) and the noisy, sparse nature of reward signals passed through text generation.

The authors ask: Can we cheaply push a model toward better alignment without running additional training loops or collecting more data?

Methods

Rather than interpolating (averaging) weights—which typically results in a performance compromise between two checkpoints—ExPO uses linear weight extrapolation:

$$\theta_2 = \theta_1 + \alpha(\theta_1 - \theta_0) = \theta_1 + \alpha \Delta\theta$$

where:

  • $\theta_0$ is the parameter set of the supervised fine-tuned (SFT) base model.
  • $\theta_1$ is the preference-aligned checkpoint (e.g., via DPO or RLHF) initialized from $\theta_0$.
  • $\alpha \ge 0$ is a scalar hyperparameter controlling the extrapolation step length.

Why does this work?

Because $\theta_1$ is initialized from $\theta_0$ and regularized to stay nearby, the weight update vector $\Delta\theta = \theta_1 - \theta_0$ acts as a "global gradient step" indicating the direction of increasing human preference. By Taylor expansion, extrapolating along $\Delta\theta$ implicitly maximizes the alignment objective via first-order approximation.

In practice, $\alpha$ is tuned like a decoding hyperparameter (searching values between $0.1$ and $0.5$) using an external reward model on a validation split, requiring only a single consumer GPU (24GB for a 7B model) for inference.

Datasets

  • Hyperparameter Search / Controlled Experiments:
  • UltraFeedback: Used for tuning $\alpha$ on its development set with an independent reward model (RM-Mistral-7B). Also used to train controlled DPO models on varying dataset fractions (5%, 10%, 20%, 40%, and 100%).
  • Evaluation Benchmarks:
  • AlpacaEval 2.0: Measures instruction-following win rates against GPT-4, utilizing the length-controlled (LC) win rate to mitigate verbosity bias.
  • MT-Bench: Multi-turn, open-ended evaluation scored from 1 to 10 by GPT-4.
  • Open LLM Leaderboard: Evaluates general reasoning and factual accuracy across ARC, HellaSwag, MMLU, GSM8K, Winogrande, and TruthfulQA.

Key Insights and Ideas

  • Extrapolation amplifies the reward signal: Alignment training identifies a viable direction in weight space toward higher rewards; ExPO simply walks further down that path without the damping effect of KL penalties.
  • Lineage is essential: ExPO strictly requires that the aligned model $\theta_1$ was directly initialized from the SFT model $\theta_0$. Extrapolating between models trained from different base checkpoints breaks mode connectivity and leads to model collapse. Similarly, extrapolating between independently trained RLHF checkpoints fails because their trajectory is non-monotonic.
  • Less data + ExPO can beat full training: In controlled tests, a model trained on only 20% of the UltraFeedback dataset boosted with ExPO achieved a 21.3% length-controlled win rate on AlpacaEval 2.0—surpassing the fully-trained baseline using 100% data (17.3%).
  • Spurious shortcuts get amplified too: If the initial alignment run overfits to spurious features (such as response length or verbosity bias), ExPO will amplify those artifacts alongside genuine preference gains.

Key Results

The authors applied ExPO to 12 popular open-source models (including Zephyr, Starling, Snorkel, Llama-3-8B-iter, InternLM2, and Tulu-2):

  • AlpacaEval 2.0: Achieved up to +4.5% length-controlled win rate gains (Tulu-2-DPO-70B: 21.2% $\to$ 25.7%) and up to +10.1% raw win rate gains (InternLM2-20B: 36.1% $\to$ 46.2%).
  • MT-Bench: Produced consistent improvements across model sizes, such as a +0.37 increase for llama3-8b-iter (8.08 $\to$ 8.45).
  • Open LLM Leaderboard: Benchmark capabilities were maintained or slightly improved (e.g., Tulu-2-DPO-70B went from 73.8 to 74.0 average score), confirming that base reasoning and knowledge are not degraded by the extrapolation.

Limitations and Future Work

  • Checkpoint Availability: ExPO requires matching SFT and alignment checkpoints from the exact same training lineage. Many releases (such as official Llama 2/3, Gemma, and Qwen weights) omit the intermediate SFT checkpoints.
  • Dependency on Reward Models: Selecting the optimal $\alpha$ currently requires evaluating outputs against an external reward model on a validation set.
  • Uniform Scalar Factor: ExPO applies the same scalar $\alpha$ uniformly across every layer and parameter tensor; layer-wise or adaptive scaling may yield better performance.
  • Application Scope: The study was conducted on text-only autoregressive Transformers. Future directions include exploring multimodal architectures (e.g., LLaVA) and alternative model families (such as state-space models like Mamba).

Summary updated

Abstract

The open-source community is experiencing a surge in the release of large language models (LLMs) that are trained to follow instructions and align with human preference. However, further training to improve them still requires expensive computational resources and data annotations. Is it possible to bypass additional training and cost-effectively acquire better-aligned models? Inspired by the literature on model interpolation, we propose a simple method called ExPO to boost LLMs' alignment with human preference. Utilizing a model that has undergone alignment training (e.g., via DPO or RLHF) and its initial SFT checkpoint, ExPO directly obtains a better-aligned model by extrapolating from the weights of the initial and the aligned models, which implicitly optimizes the alignment objective via first-order approximation. Through experiments with twelve open-source LLMs on HuggingFace, we demonstrate that ExPO consistently improves off-the-shelf DPO/RLHF models, as evaluated on the mainstream LLM benchmarks AlpacaEval 2.0 and MT-Bench. Moreover, ExPO exhibits remarkable scalability across various model sizes (from 1.8B to 70B) and capabilities. Through controlled experiments and further empirical analyses, we shed light on the essence of ExPO amplifying the reward signal learned during alignment training. Our work demonstrates the efficacy of model extrapolation in expediting the alignment of LLMs with human preference, suggesting a promising direction for future research.
BibTeX
@inproceedings{zheng2025weak,
  author = {Chujie Zheng and Ziqi Wang and Heng Ji and Minlie Huang and Nanyun Peng},
  title = {{Weak-to-Strong Extrapolation Expedites Alignment}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure