Abstract
Aligning Large Language Models (LLMs) with human preferences is a critical but computationally expensive stage of development. We introduce EXPO (Model Extrapolation), a simple, training-free technique that boosts alignment by amplifying parameter changes from short training runs. By "over-shooting" the optimization trajectory of a partially trained model, EXPO enables models trained on just 20% of the data to outperform their fully-trained counterparts, reducing computational costs by up to 75%.
Introduction
Alignment techniques like RLHF and DPO refine a model's style and behavior rather than injecting new knowledge. This stage typically involves small, incremental weight adjustments. We hypothesize that the optimization landscape for alignment is sufficiently smooth to allow for linear extrapolation, effectively jumping ahead to a more aligned state without additional backpropagation.
Methodology: Model Extrapolation (EXPO)
EXPO utilizes a first-order Taylor expansion to approximate the optimization objective. The process involves three steps:
- Initialization: Start with an initial SFT checkpoint ($M_0$) and a partially aligned model ($M_1$) trained for a limited number of steps.
- Vector Calculation: Compute the parameter difference $\Delta\theta = \theta_1 - \theta_0$.
- Extrapolation: Generate the final model $M_2$ with parameters $\theta_2 = \theta_1 + \alpha\Delta\theta$, where $\alpha$ (or $\gamma$ in some contexts) is a scaling hyperparameter.
Unlike model interpolation, which averages checkpoints, EXPO moves beyond the trained region, assuming a "straight-line" hypothesis of alignment optimization. This is justified by the observation that the parameter distance between SFT and DPO models is roughly 30 times smaller than the distance between pre-trained and SFT models.
Experimental Results
We evaluated EXPO across 12 open-source models (ranging from 1.8B to 70B parameters) using benchmarks like AlpacaEval 2.0 and MT-Bench.
Performance Gains and Efficiency
EXPO consistently improves reward scores across various model architectures and alignment algorithms (DPO, RLHF, IPO, KTO). Notably, a model trained for only 20% of its intended steps can exceed the performance of a 100%-trained model when boosted with EXPO, representing a 75% reduction in total GPU hours.
Key Insights
- Directional Quality: EXPO amplifies the existing training direction. Its success relies on the initial training steps capturing true alignment rather than spurious features (like simply increasing response length).
- Saturation Compensation: EXPO can even improve "fully trained" models, suggesting they often stop short of their peak potential alignment.
- Low Overhead: Finding the optimal extrapolation factor only requires inference-level resources, making it accessible for researchers with limited compute.
Limitations and Future Work
While effective, EXPO currently requires manual tuning of the extrapolation factor. Excessive extrapolation can lead to model collapse or "gibberish" outputs. Additionally, EXPO may slightly increase the "alignment tax," leading to minor performance dips in complex reasoning tasks like math (GSM8K). Future work could focus on automating the selection of the extrapolation factor using optimizer states.
Conclusion
EXPO provides a computationally efficient "shortcut" to model alignment. By treating alignment as a predictable trajectory in parameter space, researchers can achieve superior performance with a fraction of the traditional training budget.
Summary updated