Abstract: The pretraining-finetuning paradigm has become the prevailing trend in modern deep learning. In this work, we discover an intriguing linear phenomenon in models that are initialized from a common pretrained checkpoint and finetuned on different tasks, termed as Cross-Task Linearity (CTL). Specifically, we show that if we linearly interpolate the weights of two finetuned models, the features in the weight-interpolated model are often approximately equal to the linear interpolation of features in two finetuned models at each layer. We provide comprehensive empirical evidence supporting that CTL consistently occurs for finetuned models that start from the same pretrained checkpoint. We conjecture that in the pretraining-finetuning paradigm, neural networks approximately function as linear maps, mapping from the parameter space to the feature space. Based on this viewpoint, our study unveils novel insights into explaining model merging/editing, particularly by translating operations from the parameter space to the feature space. Furthermore, we delve deeper into the root cause for the emergence of CTL, highlighting the role of pretraining.
Introduction
The pretraining-finetuning paradigm is the cornerstone of modern artificial intelligence. Despite its ubiquity, the underlying mechanics that enable successful model merging and editing remain poorly understood. While concepts like Linear Mode Connectivity (LMC) explain why models trained on the same task can be interpolated, they do not account for the success of techniques like model averaging and task arithmetic when applied across models trained on diverse, different tasks. This paper introduces and validates Cross-Task Linearity (CTL), a fundamental linear relationship that emerges when models are finetuned from a shared pretrained base.
Methodology
Cross-Task Linearity (CTL) posits that neural networks initialized from a common pretrained checkpoint behave as linear maps between their parameter space and feature space. Formally, for two models $\theta_i$ and $\theta_j$ finetuned from the same pretrained base, CTL holds if, for any interpolation of their weights, the resulting features at each layer $\ell$ are approximately equal to the linear interpolation of the features from the original models:
$$f^{(\ell)}(\alpha\theta_i + (1-\alpha)\theta_j) \approx \alpha f^{(\ell)}(\theta_i) + (1-\alpha) f^{(\ell)}(\theta_j)$$
To validate this, the researchers measured the cosine similarity between weight-interpolated features and feature-interpolated ideals, as well as the feature length ratio ($coef_\alpha^{(\ell)}$). The study spanned multiple architectures (MLP, ResNet-18, ViT, T5) and modalities (Vision and NLP), using datasets such as ImageNet-1k, CIFAR-100, and various SQuAD/NLP benchmarks.
Empirical Evidence
The authors found that across various tasks and architectures, the cosine similarity between weight-interpolated features and feature-interpolated ideals consistently approached 1.0. This high fidelity suggests that the network's transformation remains remarkably linear in the neighborhood of the pretrained weights.
Key findings include:
- Feature-Centric Model Merging: Model averaging works because it performs a feature-level ensemble at every layer of the network, rather than just at the output.
- Task Arithmetic as Aggregation: Adding "task vectors" in the parameter space is equivalent to aggregating task-specific information in the feature space. This explains how adding a specific capability (e.g., a new language) can be achieved without destroying the base model's original knowledge.
- Pretraining as the Catalyst: CTL does not appear in randomly initialized models. It emerges specifically from pretraining, which flattens the loss landscape and constrains finetuned weights to a region where linear approximations are valid.
Discussion and Implications
The discovery of CTL provides a unified, feature-centric explanation for why popular model-merging techniques are effective. It suggests that in the pretraining-finetuning regime, complex neural networks effectively function as linear operators mapping parameters directly to feature space. This insight allows for targeted model editing, such as "forgetting" specific tasks by negating their task vectors, which effectively resets that portion of the feature space to its pretrained state.
While this work provides a significant step toward a theoretical understanding of model interpolation, further research is needed to develop a more rigorous mathematical framework and to explicitly test these principles on massive decoder-only Large Language Models (LLMs).
Conclusion
Cross-Task Linearity reveals a hidden simplicity in the way finetuned models relate to one another. By demonstrating that parameter-space operations translate linearly to feature-space operations, this work provides both a theoretical foundation for current model-merging practices and a roadmap for more precise model editing in the future.
Summary updated