Fine-tuning can cripple foundation models; preserving features may be the solution
Its authors withdrew this paper, so there is no full text to read or summarize.
Lists
Only you see this
Abstract
Pre-trained foundation models, due to their enormous capacity and their training using vast amounts of data can store knowledge about many real-world concepts. To further improve performance on downstream tasks, these models can be fine-tuned on task specific datasets. While various fine-tuning methods have been devised and have been shown to be highly effective, we observe that a fine-tuned model's ability to recognize concepts on tasks different from the downstream one is reduced significantly compared to its pre-trained counterpart. This is clearly undesirable as a huge amount of time and money went into learning those very concepts in the first place. We call this undesirable phenomenon "concept forgetting'' and via experiments show that most end-to-end fine-tuning approaches suffer heavily from this side effect. To this end, we also propose a rather simple fix to this problem by designing a method called LDIFS (short for $\ell_2$ distance in feature space) that simply preserves the features of the original foundation model during fine-tuning. We show that LDIFS significantly reduces concept forgetting without having noticeable impact on the downstream task performance.
BibTeX
@article{mukhoti2024fine,
author = {Jishnu Mukhoti and Yarin Gal and Philip Torr and Puneet K. Dokania},
title = {{Fine-tuning can cripple foundation models; preserving features may be the solution}},
journal = {Accepted by TMLR},
year = {2024},
url = {https://openreview.net/forum?id=kfhoeZCeW7},
}