Weak-to-Strong Extrapolation Expedites Alignment

Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, Nanyun Peng

Abstract

Aligning Large Language Models (LLMs) with human preferences is a critical but computationally expensive stage of development. We introduce EXPO (Model Extrapolation), a simple, training-free technique that boosts alignment by amplifying parameter changes from short training runs. By "over-shooting" the optimization trajectory of a partially trained model, EXPO enables models trained on just 20% of the data to outperform their fully-trained counterparts, reducing computational costs by up to 75%.

Introduction

Alignment techniques like RLHF and DPO refine a model's style and behavior rather than injecting new knowledge. This stage typically involves small, incremental weight adjustments. We hypothesize that the optimization landscape for alignment is sufficiently smooth to allow for linear extrapolation, effectively jumping ahead to a more aligned state without additional backpropagation.

Methodology: Model Extrapolation (EXPO)

EXPO utilizes a first-order Taylor expansion to approximate the optimization objective. The process involves three steps:

  1. Initialization: Start with an initial SFT checkpoint ($M_0$) and a partially aligned model ($M_1$) trained for a limited number of steps.
  2. Vector Calculation: Compute the parameter difference $\Delta\theta = \theta_1 - \theta_0$.
  3. Extrapolation: Generate the final model $M_2$ with parameters $\theta_2 = \theta_1 + \alpha\Delta\theta$, where $\alpha$ (or $\gamma$ in some contexts) is a scaling hyperparameter.
Conceptual diagram of EXPO: extrapolating from the initial state $\theta_0$ through the partially trained state $\theta_1$ to reach a more aligned state $\theta_2$.

Unlike model interpolation, which averages checkpoints, EXPO moves beyond the trained region, assuming a "straight-line" hypothesis of alignment optimization. This is justified by the observation that the parameter distance between SFT and DPO models is roughly 30 times smaller than the distance between pre-trained and SFT models.

Experimental Results

We evaluated EXPO across 12 open-source models (ranging from 1.8B to 70B parameters) using benchmarks like AlpacaEval 2.0 and MT-Bench.

Performance Gains and Efficiency

EXPO consistently improves reward scores across various model architectures and alignment algorithms (DPO, RLHF, IPO, KTO). Notably, a model trained for only 20% of its intended steps can exceed the performance of a 100%-trained model when boosted with EXPO, representing a 75% reduction in total GPU hours.

Reward scores for multiple models (Tulu, InternLM, Starling) as the extrapolation factor increases, demonstrating consistent alignment gains across different base models.

Key Insights

  • Directional Quality: EXPO amplifies the existing training direction. Its success relies on the initial training steps capturing true alignment rather than spurious features (like simply increasing response length).
  • Saturation Compensation: EXPO can even improve "fully trained" models, suggesting they often stop short of their peak potential alignment.
  • Low Overhead: Finding the optimal extrapolation factor only requires inference-level resources, making it accessible for researchers with limited compute.

Limitations and Future Work

While effective, EXPO currently requires manual tuning of the extrapolation factor. Excessive extrapolation can lead to model collapse or "gibberish" outputs. Additionally, EXPO may slightly increase the "alignment tax," leading to minor performance dips in complex reasoning tasks like math (GSM8K). Future work could focus on automating the selection of the extrapolation factor using optimizer states.

Conclusion

EXPO provides a computationally efficient "shortcut" to model alignment. By treating alignment as a predictable trajectory in parameter space, researchers can achieve superior performance with a fraction of the traditional training budget.

Summary updated

BibTeX
@inproceedings{zheng2025weak,
  author = {Chujie Zheng and Ziqi Wang and Heng Ji and Minlie Huang and Nanyun Peng},
  title = {{Weak-to-Strong Extrapolation Expedites Alignment}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure