TL;DR
Aligning large language models with human preference via RLHF or DPO is computationally expensive and often results in undertrained alignment due to regularization constraints. ExPO (Weak-to-Strong Extrapolation) bypasses further training by simply extrapolating model weights along the direction from the initial SFT checkpoint to the aligned checkpoint ($\theta_2 = \theta_1 + \alpha(\theta_1 - \theta_0)$). This zero-training trick consistently boosts instruction-following and preference alignment across 12 open models ranging from 1.8B to 70B parameters.
What Problem Does This Paper Address?
Post-training large language models (LLMs) to match human preferences via reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO) requires substantial compute and preference annotations. Even after alignment, most open-source models remain suboptimally aligned. This under-optimization happens partly because of strict regularization terms (like the KL-divergence penalty that prevents models from drifting too far from the reference model) and the noisy, sparse nature of reward signals passed through text generation.
The authors ask: Can we cheaply push a model toward better alignment without running additional training loops or collecting more data?
Methods
Rather than interpolating (averaging) weights—which typically results in a performance compromise between two checkpoints—ExPO uses linear weight extrapolation:
$$\theta_2 = \theta_1 + \alpha(\theta_1 - \theta_0) = \theta_1 + \alpha \Delta\theta$$
where:
- $\theta_0$ is the parameter set of the supervised fine-tuned (SFT) base model.
- $\theta_1$ is the preference-aligned checkpoint (e.g., via DPO or RLHF) initialized from $\theta_0$.
- $\alpha \ge 0$ is a scalar hyperparameter controlling the extrapolation step length.
Why does this work?
Because $\theta_1$ is initialized from $\theta_0$ and regularized to stay nearby, the weight update vector $\Delta\theta = \theta_1 - \theta_0$ acts as a "global gradient step" indicating the direction of increasing human preference. By Taylor expansion, extrapolating along $\Delta\theta$ implicitly maximizes the alignment objective via first-order approximation.
In practice, $\alpha$ is tuned like a decoding hyperparameter (searching values between $0.1$ and $0.5$) using an external reward model on a validation split, requiring only a single consumer GPU (24GB for a 7B model) for inference.
Datasets
- Hyperparameter Search / Controlled Experiments:
- UltraFeedback: Used for tuning $\alpha$ on its development set with an independent reward model (
RM-Mistral-7B). Also used to train controlled DPO models on varying dataset fractions (5%, 10%, 20%, 40%, and 100%). - Evaluation Benchmarks:
- AlpacaEval 2.0: Measures instruction-following win rates against GPT-4, utilizing the length-controlled (LC) win rate to mitigate verbosity bias.
- MT-Bench: Multi-turn, open-ended evaluation scored from 1 to 10 by GPT-4.
- Open LLM Leaderboard: Evaluates general reasoning and factual accuracy across ARC, HellaSwag, MMLU, GSM8K, Winogrande, and TruthfulQA.
Key Insights and Ideas
- Extrapolation amplifies the reward signal: Alignment training identifies a viable direction in weight space toward higher rewards; ExPO simply walks further down that path without the damping effect of KL penalties.
- Lineage is essential: ExPO strictly requires that the aligned model $\theta_1$ was directly initialized from the SFT model $\theta_0$. Extrapolating between models trained from different base checkpoints breaks mode connectivity and leads to model collapse. Similarly, extrapolating between independently trained RLHF checkpoints fails because their trajectory is non-monotonic.
- Less data + ExPO can beat full training: In controlled tests, a model trained on only 20% of the UltraFeedback dataset boosted with ExPO achieved a 21.3% length-controlled win rate on AlpacaEval 2.0—surpassing the fully-trained baseline using 100% data (17.3%).
- Spurious shortcuts get amplified too: If the initial alignment run overfits to spurious features (such as response length or verbosity bias), ExPO will amplify those artifacts alongside genuine preference gains.
Key Results
The authors applied ExPO to 12 popular open-source models (including Zephyr, Starling, Snorkel, Llama-3-8B-iter, InternLM2, and Tulu-2):
- AlpacaEval 2.0: Achieved up to +4.5% length-controlled win rate gains (Tulu-2-DPO-70B: 21.2% $\to$ 25.7%) and up to +10.1% raw win rate gains (InternLM2-20B: 36.1% $\to$ 46.2%).
- MT-Bench: Produced consistent improvements across model sizes, such as a +0.37 increase for
llama3-8b-iter(8.08 $\to$ 8.45). - Open LLM Leaderboard: Benchmark capabilities were maintained or slightly improved (e.g., Tulu-2-DPO-70B went from 73.8 to 74.0 average score), confirming that base reasoning and knowledge are not degraded by the extrapolation.
Limitations and Future Work
- Checkpoint Availability: ExPO requires matching SFT and alignment checkpoints from the exact same training lineage. Many releases (such as official Llama 2/3, Gemma, and Qwen weights) omit the intermediate SFT checkpoints.
- Dependency on Reward Models: Selecting the optimal $\alpha$ currently requires evaluating outputs against an external reward model on a validation set.
- Uniform Scalar Factor: ExPO applies the same scalar $\alpha$ uniformly across every layer and parameter tensor; layer-wise or adaptive scaling may yield better performance.
- Application Scope: The study was conducted on text-only autoregressive Transformers. Future directions include exploring multimodal architectures (e.g., LLaVA) and alternative model families (such as state-space models like Mamba).
Summary updated