Update Your Transformer to the Latest Release: Re-Basin of Task Vectors

Filippo Rinaldi, Giacomo Capitani, Lorenzo Bonicelli, Donato Crisostomi, Federico Bolelli, Elisa Ficarra, Emanuele Rodolà, Simone Calderara, Angelo Porrello

TL;DR: TransFusion is a novel, data-free method that allows researchers to port fine-tuned knowledge—stored as task vectors—from an older Transformer model to a newly released checkpoint. By using a two-level weight permutation strategy rooted in spectral theory, it aligns the internal representations of different models without requiring a single step of retraining or access to the original training data.

The Foundation Model Upgrade Problem

In the rapidly evolving landscape of artificial intelligence, foundation models like CLIP and BERT are frequently updated by their creators. These new releases typically retain the same architecture but are trained on larger, more curated datasets, yielding superior zero-shot performance.

However, this rapid progress creates a significant headache for practitioners. If you have spent weeks fine-tuning a specific version of a model for a niche task—such as medical imaging or satellite analysis—your specialized model becomes obsolete the moment a better base model is released. Traditionally, the only way to upgrade is to retrain the new base model on your downstream data from scratch. This is computationally expensive and sometimes impossible if the original training data is no longer available due to privacy regulations or storage constraints.

Why can't we just copy the learned weights from the old model to the new one? Because the two models, despite sharing an architecture, live in different "basins" of the loss landscape. Their internal neurons are ordered differently, making direct weight transfer ineffective.

TransFusion: A Data-Free Solution

To solve this, the authors introduce TransFusion, a method based on "model re-basin"—the concept of permuting the units (neurons) of one model to align its internal structure with another. While previous re-basin methods worked for simpler networks, they failed on Transformers due to the complexity of Multi-Head Attention (MHA) and residual connections.

TransFusion overcomes these hurdles by permuting the task vector (the mathematical difference between the fine-tuned and pre-trained weights) and adding it to the new base model. It achieves this alignment through three key innovations:

  • Inter-Head Alignment via Spectral Fingerprinting: You cannot treat a Transformer layer as one giant matrix; you must respect the boundaries of attention heads. Using spectral theory, TransFusion calculates the singular values (via SVD) of each attention head. Because singular values are invariant to the internal ordering of neurons, they act as a unique "fingerprint." The method uses these fingerprints to match a head in the old model to the most similar head in the new model.
  • Intra-Head Alignment: Once the heads are matched, TransFusion uses the Hungarian algorithm to permute the individual neurons within those heads to maximize similarity. This prevents "head contamination," ensuring neurons from different logical heads do not get mixed.
  • Residual Path Management: Transformers rely heavily on residual connections. TransFusion ensures that both the main network path and the shortcut path undergo identical permutations, maintaining the functional integrity of the model.

Key Results and Performance

The authors evaluated TransFusion across both vision (EuroSAT, DTD, GTSRB, SVHN) and language domains (GLUE benchmarks like QQP, SST2, RTE, CoLA), using CLIP ViT-B/16 models pre-trained on different datasets.

  • Superior Accuracy: On the SVHN vision task, TransFusion improved the new base model's accuracy by +3.64%. In stark contrast, a "vanilla" transport (adding the task vector without permutation) caused a disastrous -22% drop in performance.
  • NLP Gains: On the QQP text task, TransFusion achieved a +6.50% gain over the base model, significantly outperforming previous state-of-the-art re-basin methods like Git Re-Basin.
  • Functional Equivalence: Unlike competing methods such as Optimal Transport (OT), TransFusion preserves the mathematical functionality of the original model.
  • Computational Efficiency: The process scales polynomially with model size ($O(L d_m^3)$), making it exponentially faster than even a few epochs of retraining.

Limitations and Future Directions

While TransFusion proves that data-free transfer is viable, it has a few limitations. First, the method's success is heavily dependent on the quality of the original task vector. For instance, on the DTD dataset—which had very few training examples (40 per class)—the transfer was less effective.

Additionally, TransFusion currently requires the two models to have the exact same architecture (identical number of layers and heads). Future work could explore "pruning" or "replicating" layers to allow knowledge transfer between models of different sizes, such as upgrading from a ViT-Base to a ViT-Large. Finally, while the data-free approach is highly capable, incorporating a tiny "support set" of data could likely refine the alignment even further in hybrid approaches.

Summary updated

Abstract

Foundation models serve as the backbone for numerous specialized models developed through fine-tuning. However, when the underlying pretrained model is updated or retrained (e.g., on larger and more curated datasets), the fine-tuned model becomes obsolete, losing its utility and requiring retraining. This raises the question: is it possible to transfer fine-tuning to a new release of the model? In this work, we investigate how to transfer fine-tuning to a new checkpoint without having to re-train, in a data-free manner. To do so, we draw principles from model re-basin and provide a recipe based on weight permutations to re-base the modifications made to the original base model, often called task vector. In particular, our approach tailors model re-basin for Transformer models, taking into account the challenges of residual connections and multi-head attention layers. Specifically, we propose a two-level method rooted in spectral theory, initially permuting the attention heads and subsequently adjusting parameters within select pairs of heads. Through extensive experiments on visual and textual tasks, we achieve the seamless transfer of fine-tuned knowledge to new pre-trained backbones without relying on a single training step or datapoint. Code is available at https://github.com/aimagelab/TransFusion.
BibTeX
@inproceedings{rinaldi2025update,
  author = {Filippo Rinaldi and Giacomo Capitani and Lorenzo Bonicelli and Donato Crisostomi and Federico Bolelli and Elisa Ficarra and Emanuele Rodolà and Simone Calderara and Angelo Porrello},
  title = {{Update Your Transformer to the Latest Release: Re-Basin of Task Vectors}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=sHvImzN9pL},
}
Figure