Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgetting

Haolin Chen, Philip N. Garner

Introduction

Fine-tuning large-scale pre-trained models has become the standard paradigm for adapting general-purpose AI to specialized tasks. However, this process frequently suffers from catastrophic forgetting, where the model's new specialized knowledge (e.g., a specific speaker's accent in a Text-to-Speech system) comes at the expense of its original general-purpose capabilities. While Parameter-Efficient Fine-Tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) significantly reduce the number of trainable parameters, they do not inherently prevent the overwriting of pre-trained knowledge.

This paper introduces a principled Bayesian framework to address this challenge. By applying Kronecker-factored Laplace approximations to PEFT methods, the authors demonstrate that models can acquire new skills while maintaining a high degree of fidelity to their original pre-training.

Methodology: The Bayesian Framework

The authors frame fine-tuning as a Maximum A Posteriori (MAP) estimation problem. The core idea is to represent the pre-trained model's knowledge as a Gaussian distribution using the Laplace approximation. During fine-tuning, the model is regularized to stay "close" to the pre-trained weights, with the "closeness" weighted by the importance of each parameter. This importance is quantified by the Fisher Information Matrix (FIM).

The study compares three distinct FIM approximations applied to the LoRA "delta" weights:

  1. L2-SP: A uniform penalty that treats all parameters as equally important.
  2. Elastic Weight Consolidation (EWC): A diagonal approximation assuming parameter independence.
  3. Kronecker-Factored Approximate Curvature (KFAC): A sophisticated approach that captures parameter interactions within the same layer.

As long as the parameter update is differentiable, these Bayesian penalties can be integrated directly into the standard fine-tuning loss function.

Experimental Results

Language Modeling and Classification

The approach was evaluated using the OPT family of models (350M and 1.3B parameters) across the GLUE benchmark and WikiText datasets. A critical finding was the severe forgetting exhibited by unregularized LoRA. For the OPT-350M model fine-tuned on MNLI, the perplexity on pre-training data (WikiText-103) surged from an initial 15.40 to 523.7, indicating a near-total loss of linguistic coherence. In contrast, KFAC regularization maintained a perplexity of 17.24, nearly matching the original pre-trained performance.

Comparative performance of regularization methods on OPT-350M and OPT-1.3B across classification (Accuracy) and language modeling (Perplexity) tasks.

Speech Synthesis (TTS)

In the domain of speech synthesis using StyleTTS 2, the authors tested speaker adaptation. While standard LoRA successfully adapted to a target Indian accent, it caused a significant performance drop for other, out-of-domain (OOD) speakers. KFAC-regularized models preserved OOD similarity scores (0.280) much more effectively than EWC (0.224) or standard LoRA (0.203), maintaining the model's versatility.

Speaker adaptation results for StyleTTS 2, comparing target speaker similarity and out-of-domain (OOD) speaker preservation.

Key Insights and Analysis

  • The Importance of Interactions: KFAC significantly outperforms diagonal methods like EWC. By accounting for how weights within a layer interact, KFAC provides a more precise map of the model's existing knowledge, allowing for more flexible updates that do not disrupt established features.
  • The Scale Shield: Larger models (e.g., OPT-1.3B) show greater inherent resistance to forgetting than smaller models, though Bayesian regularization still provides a measurable performance boost across all scales.
  • Data Efficiency: While KFAC is superior in accuracy, it is more sensitive to the quantity of data used to estimate the Hessian. EWC remains a robust alternative when only a very small number of pre-training samples are available.

Conclusion and Limitations

Bayesian PEFT offers a powerful tool for maintaining model versatility during adaptation. However, the method requires access to a subset of the original pre-training data to calculate the FIM, which may not be possible for all proprietary models. Additionally, KFAC introduces higher computational and memory overhead during the initial estimation phase compared to simpler diagonal approximations. Future work will explore applying these methods to additive PEFT techniques and scaling to even larger models.

Summary updated

BibTeX
@article{chen2024bayesian,
  author = {Haolin Chen and Philip N. Garner},
  title = {{Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgetting}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2024},
  doi = {10.1109/taslp.2024.3463395},
}
Figure