TL;DR
Post-training fine-tunes large foundation models into specialized versions, but modifying the resulting parameter shifts ("delta parameters") through pruning, quantization, or extrapolation has largely relied on disparate heuristics. This paper introduces a unified theoretical framework using Riemann sum approximations of the loss function, explaining why specific edits preserve, degrade, or enhance model performance. Leveraging this perspective, the authors generalize popular techniques like DARE, BitDelta, and EXPO.
What Problem Does This Paper Address?
When adapting large pre-trained base models ($W_{\text{PRE}}$) to task-specific checkpoints ($W_{\text{POST}}$), all learned adjustments are captured by the delta parameters: $$\Delta W = W_{\text{POST}} - W_{\text{PRE}}$$
Recent research has developed various techniques to edit these delta parameters—such as random pruning and rescaling (DARE), 1-bit quantization (BitDelta), low-rank SVD (Twin-Merging), magnitude pruning (TIES-Merging), and weight extrapolation (EXPO). However, the community lacked a unifying mathematical framework to explain why these disparate operations affect loss and performance so differently.
Methods
The authors analyze parameter perturbations $\Delta\tilde{W}$ using a first-order Riemann sum approximation of the resulting loss difference:
$$\Delta \mathcal{L} = \mathcal{L}(W_{\text{POST}} + \Delta\tilde{W}) - \mathcal{L}(W_{\text{POST}}) \approx \frac{1}{C}\sum_{c=0}^{C-1} \langle \nabla \mathcal{L}^c, \Delta\tilde{W} \rangle$$
Depending on the sign of this approximation term, editing methods fall into three distinct regimes:
- Competitive Performance ($\Delta \mathcal{L} \approx 0$): In methods like DARE, the random dropout and uniform rescaling cancel out in expectation across the parameters, driving the inner-product term to zero. The authors generalize DARE by introducing a retention scaling factor $k$ (where DARE is the special case $k=0$), proving that any affine rescaling preserving the mean perturbation leaves the expected loss unchanged.
- Decreased Performance ($\Delta \mathcal{L} > 0$): Quantization (BitDelta), SVD truncation (Twin-Merging), and magnitude pruning (TIES-Merging) introduce systematic biases whose inner product with the gradient is positive, slightly increasing the task loss.
- Improved Performance ($\Delta \mathcal{L} < 0$): Because post-training (e.g., DPO/RLHF) is often suboptimal, the negative gradient still points in the direction of the fine-tuning delta. Extrapolating or interpolating along $\Delta W$ creates a negative approximation term, lowering the loss.
Datasets
The authors validate their framework across vision and language benchmarks using LLaMA-3-8B-Instruct, Mistral-7B-Instruct-v0.3, Zephyr-7B, and ViT-B-32:
- Language Benchmarks:
- 25-shot ARC Challenge
- 5-shot GSM8K
- 10-shot HellaSwag
- Zero-shot HumanEval
- Zero-shot IFEval
- 5-shot MMLU
- Zero-shot TruthfulQA
- Zero-shot Winogrande
- Preference alignment loss evaluated on UltraFeedback
- Vision Benchmarks: DTD, EuroSAT, and GTSRB.
No new datasets were introduced.
Key Insights and Ideas
- The real invariance in DARE is gradient-alignment randomness: Prior work hypothesized that DARE succeeds by approximating original activation embeddings. This paper shows that random dropout over delta parameters is sufficient but not necessary; what is both necessary and sufficient is that the element-wise product of parameter updates and gradients ($\Delta W \odot \nabla \mathcal{L}$) remains unbiased. Artificially biasing $\Delta W$ preserves GSM8K performance (~74–76%), but biasing $\Delta W \odot \nabla \mathcal{L}$ collapses accuracy to 0%.
- Extrapolation is not a silver bullet: EXPO's success does not inherently stem from extrapolation ($\alpha > 0$). Whether extrapolation or interpolation ($\alpha < 0$) improves performance depends strictly on the sign of the local loss gradient relative to $\Delta W$.
- BitDelta expands cleanly to multi-bit: Representing delta parameters with clustered magnitude blocks (rather than a single global magnitude scalar) scales monotonically with the number of bits.
Key Results
- Generalized DARE: Across GSM8K, TruthfulQA, HumanEval, and vision tasks (EuroSAT, DTD, GTSRB), adjusting non-zero $k$ values (e.g., $k \in [-1.0, 0.8]$) consistently maintained accuracy parity with the post-trained baseline, verifying the generalized formulation.
- Multi-bit BitDelta: Scaling BitDelta from 1-bit ($\approx 50\%$ accuracy on GSM8K) to 4-bit representations surpassed the uncompressed post-trained model performance on both LLaMA-3-8B-Instruct and Mistral-7B-Instruct.
- Interpolation vs. Extrapolation: On LLaMA-3-8B-Instruct, interpolation actually outperformed extrapolation on 7 out of 8 evaluated benchmarks (ARC Challenge, GSM8K, HellaSwag, MMLU, TruthfulQA, Winogrande). Extrapolation provided substantial gains solely on IFEval (+15 point gap).
Limitations and Future Work
- Instruction-following anomalies: In the BitDelta noise ablation, IFEval behaved anomalously compared to GSM8K, which the authors attribute to the unique dynamics of instruction-following tasks and leave for future study.
- Local linear approximation limits: The Riemann sum framework relies on gradients remaining locally informative in the neighborhood of $W_{\text{POST}}$. It breaks down when step sizes ($\alpha$ or drop rates $p$) become excessively large.
- Future directions: The authors suggest using this framework to guide lossless 1–4 bit model quantization, data-free model enhancement by directly minimizing the Riemann approximation term, and post-training interpretability by pruning deltas while preserving loss invariance.
Summary updated