Abstract
Model merging enables efficient multi-task models by combining task-specific fine-tuned checkpoints. However, storing multiple task-specific checkpoints requires significant memory, limiting scalability and restricting model merging to larger models and diverse tasks. In this paper, we propose quantizing task vectors (i.e., the difference between pre-trained and fine-tuned checkpoints) instead of quantizing fine-tuned checkpoints. We observe that task vectors exhibit a narrow weight range, enabling low precision quantization (up to 4 bit) within existing task vector merging frameworks. To further mitigate quantization errors within ultra-low bit precision (e.g., 2 bit), we introduce Residual Task Vector Quantization, which decomposes the task vector into a base vector and offset component. We allocate bits based on quantization sensitivity, ensuring precision while minimizing error within a memory budget. Experiments on image classification and dense prediction show our method maintains or improves model merging performance while using only 8% of the memory required for full-precision checkpoints.
Introduction
Model merging has emerged as a powerful paradigm for creating multi-task models by combining specialized, fine-tuned checkpoints derived from a common pre-trained base. Techniques like Task Arithmetic allow for the linear combination of these weights, enabling a single model to perform diverse tasks without the prohibitive cost of joint training. However, a significant practical hurdle remains: storage. Storing multiple full-precision (FP32) checkpoints for dozens of tasks requires massive memory, often exceeding the capacity of edge devices and limiting the scalability of multi-task systems. Standard quantization techniques, while effective for single models, often degrade the specific task-related information necessary for effective merging.
Proposed Methods
To address the storage bottleneck, we propose Task Vector Quantization (TVQ). Instead of quantizing the complete fine-tuned model weights ($\theta_{ft}$), TVQ focuses on the task vector ($\tau$), defined as the difference between the fine-tuned weights and the original pre-trained weights ($\tau = \theta_{ft} - \theta_{pre}$).
As shown in Figure 1, while prior work stores full FP32 checkpoints, TVQ reduces the task-specific storage to INT8 or lower. We observe that task vectors naturally exhibit a much narrower range of values compared to full weights, making them highly amenable to low-precision quantization (e.g., 3 or 4 bits) with minimal information loss. This "narrow range" advantage allows the model to retain delicate task-specific information that standard quantization often destroys.
For scenarios requiring ultra-low bit precision (e.g., 2 bits), we introduce Residual Task Vector Quantization (RTVQ). RTVQ decomposes the task vector into a shared, high-precision base vector (the average across tasks) and task-specific low-precision offset vectors. By allocating more bits to the shared component and fewer to the offsets, RTVQ maintains high accuracy within a strict memory budget. A quantization error correction step is also employed to calibrate the base vector, preventing rounding errors from compounding during the offset calculation.
Experimental Setup
The effectiveness of TVQ and RTVQ was evaluated across three diverse domains:
- Image Classification: A suite of 19 datasets (e.g., SUN397, EuroSAT) with scalability tests up to 20 tasks.
- Dense Vision Tasks: Using the NYUv2 dataset for semantic segmentation, depth estimation, and surface normal estimation.
- Natural Language Processing (NLP): Evaluated on the GLUE benchmark, covering tasks like sentiment analysis and linguistic acceptability.
Key Results and Discussion
Our experiments demonstrate that TVQ can achieve a 16x reduction in storage for a ViT-L model—reducing the storage for 20 tasks from 22.8 GB to just 1.4 GB at 2-bit precision—while maintaining competitive performance.
As illustrated in Figure 2, 3-bit TVQ often matches or even exceeds the performance of the FP32 baseline in classification tasks. This suggests that quantization may act as a form of regularization or denoising, preventing overfitting to task-specific noise and improving generalization. Furthermore, quantization appears to increase the orthogonality of task vectors, reducing task interference during the merge.
While dense tasks are more sensitive to quantization noise, RTVQ significantly outperforms standard TVQ at the 2-bit level, proving robust even in spatially precise domains. In some cases, 2-bit quantization actually improved cross-task accuracy from 38.7% to 48.4%, indicating better knowledge sharing between tasks facilitated by the compression process.
Conclusion
Task Vector Quantization provides a scalable and memory-efficient framework for model merging. By exploiting the narrow range of task vectors and employing residual decomposition, we can compress task-specific information by over 90% without sacrificing performance. This paves the way for deploying sophisticated multi-task models on resource-constrained hardware and scaling model merging to hundreds of tasks. Future work will explore the application of these techniques to large-scale generative models (LLMs) and more complex merging strategies.
Summary updated