A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models

Qiaoyu Tang, Le Yu, Bowen Yu, Hongyu Lin, Keming Lu, Yaojie Lu, Xianpei Han, Le SunSubmission to ICLR 2025 · Withdrawn

TL;DR

Post-training fine-tunes large foundation models into specialized versions, but modifying the resulting parameter shifts ("delta parameters") through pruning, quantization, or extrapolation has largely relied on disparate heuristics. This paper introduces a unified theoretical framework using Riemann sum approximations of the loss function, explaining why specific edits preserve, degrade, or enhance model performance. Leveraging this perspective, the authors generalize popular techniques like DARE, BitDelta, and EXPO.

What Problem Does This Paper Address?

When adapting large pre-trained base models ($W_{\text{PRE}}$) to task-specific checkpoints ($W_{\text{POST}}$), all learned adjustments are captured by the delta parameters: $$\Delta W = W_{\text{POST}} - W_{\text{PRE}}$$

Recent research has developed various techniques to edit these delta parameters—such as random pruning and rescaling (DARE), 1-bit quantization (BitDelta), low-rank SVD (Twin-Merging), magnitude pruning (TIES-Merging), and weight extrapolation (EXPO). However, the community lacked a unifying mathematical framework to explain why these disparate operations affect loss and performance so differently.

Methods

The authors analyze parameter perturbations $\Delta\tilde{W}$ using a first-order Riemann sum approximation of the resulting loss difference:

$$\Delta \mathcal{L} = \mathcal{L}(W_{\text{POST}} + \Delta\tilde{W}) - \mathcal{L}(W_{\text{POST}}) \approx \frac{1}{C}\sum_{c=0}^{C-1} \langle \nabla \mathcal{L}^c, \Delta\tilde{W} \rangle$$

Depending on the sign of this approximation term, editing methods fall into three distinct regimes:

  1. Competitive Performance ($\Delta \mathcal{L} \approx 0$): In methods like DARE, the random dropout and uniform rescaling cancel out in expectation across the parameters, driving the inner-product term to zero. The authors generalize DARE by introducing a retention scaling factor $k$ (where DARE is the special case $k=0$), proving that any affine rescaling preserving the mean perturbation leaves the expected loss unchanged.
  2. Decreased Performance ($\Delta \mathcal{L} > 0$): Quantization (BitDelta), SVD truncation (Twin-Merging), and magnitude pruning (TIES-Merging) introduce systematic biases whose inner product with the gradient is positive, slightly increasing the task loss.
  3. Improved Performance ($\Delta \mathcal{L} < 0$): Because post-training (e.g., DPO/RLHF) is often suboptimal, the negative gradient still points in the direction of the fine-tuning delta. Extrapolating or interpolating along $\Delta W$ creates a negative approximation term, lowering the loss.

Datasets

The authors validate their framework across vision and language benchmarks using LLaMA-3-8B-Instruct, Mistral-7B-Instruct-v0.3, Zephyr-7B, and ViT-B-32:

  • Language Benchmarks:
  • 25-shot ARC Challenge
  • 5-shot GSM8K
  • 10-shot HellaSwag
  • Zero-shot HumanEval
  • Zero-shot IFEval
  • 5-shot MMLU
  • Zero-shot TruthfulQA
  • Zero-shot Winogrande
  • Preference alignment loss evaluated on UltraFeedback
  • Vision Benchmarks: DTD, EuroSAT, and GTSRB.

No new datasets were introduced.

Key Insights and Ideas

  • The real invariance in DARE is gradient-alignment randomness: Prior work hypothesized that DARE succeeds by approximating original activation embeddings. This paper shows that random dropout over delta parameters is sufficient but not necessary; what is both necessary and sufficient is that the element-wise product of parameter updates and gradients ($\Delta W \odot \nabla \mathcal{L}$) remains unbiased. Artificially biasing $\Delta W$ preserves GSM8K performance (~74–76%), but biasing $\Delta W \odot \nabla \mathcal{L}$ collapses accuracy to 0%.
  • Extrapolation is not a silver bullet: EXPO's success does not inherently stem from extrapolation ($\alpha > 0$). Whether extrapolation or interpolation ($\alpha < 0$) improves performance depends strictly on the sign of the local loss gradient relative to $\Delta W$.
  • BitDelta expands cleanly to multi-bit: Representing delta parameters with clustered magnitude blocks (rather than a single global magnitude scalar) scales monotonically with the number of bits.

Key Results

  • Generalized DARE: Across GSM8K, TruthfulQA, HumanEval, and vision tasks (EuroSAT, DTD, GTSRB), adjusting non-zero $k$ values (e.g., $k \in [-1.0, 0.8]$) consistently maintained accuracy parity with the post-trained baseline, verifying the generalized formulation.
  • Multi-bit BitDelta: Scaling BitDelta from 1-bit ($\approx 50\%$ accuracy on GSM8K) to 4-bit representations surpassed the uncompressed post-trained model performance on both LLaMA-3-8B-Instruct and Mistral-7B-Instruct.
  • Interpolation vs. Extrapolation: On LLaMA-3-8B-Instruct, interpolation actually outperformed extrapolation on 7 out of 8 evaluated benchmarks (ARC Challenge, GSM8K, HellaSwag, MMLU, TruthfulQA, Winogrande). Extrapolation provided substantial gains solely on IFEval (+15 point gap).

Limitations and Future Work

  • Instruction-following anomalies: In the BitDelta noise ablation, IFEval behaved anomalously compared to GSM8K, which the authors attribute to the unique dynamics of instruction-following tasks and leave for future study.
  • Local linear approximation limits: The Riemann sum framework relies on gradients remaining locally informative in the neighborhood of $W_{\text{POST}}$. It breaks down when step sizes ($\alpha$ or drop rates $p$) become excessively large.
  • Future directions: The authors suggest using this framework to guide lossless 1–4 bit model quantization, data-free model enhancement by directly minimizing the Riemann approximation term, and post-training interpretability by pruning deltas while preserving loss invariance.

Summary updated

Abstract

Post-training has emerged as a crucial paradigm for adapting large-scale pre-trained models to various tasks, whose effects are fully reflected by delta parameters (i.e., the disparity between post-trained and pre-trained parameters). While numerous studies have explored delta parameter properties via operations like pruning, quantization, low-rank approximation, and extrapolation, a unified framework for systematically examining these characteristics has been lacking. In this paper, we propose a novel perspective based on Riemann sum approximation of the loss function to elucidate delta parameter editing operations. Our analysis categorizes existing methods into three classes based on their post-editing performance: competitive, decreased, and improved, explaining how they are expressed by the Riemann sum approximation term and how they alter the model performance. Extensive experiments on both visual and language models, including ViT, LLaMA 3, and Mistral, corroborate our theoretical findings. Furthermore, we introduce extensions to existing techniques like DARE and BitDelta, highlighting their limitations in leveraging the properties of delta parameters and reorganizing them into general expressions to enhance the applicability and effectiveness of delta parameter editing in post-trained models.
BibTeX
@misc{tang2024unified,
  author = {Qiaoyu Tang and Le Yu and Bowen Yu and Hongyu Lin and Keming Lu and Yaojie Lu and Xianpei Han and Le Sun},
  title = {{A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models}},
  year = {2024},
  eprint = {2410.13841},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.13841},
}
Figure