Knowledge-Adaptation Priors

Mohammad Emtiyaz Khan, Siddharth Swaroop

Abstract

Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch. We present Knowledge-adaptation priors (K-priors) to reduce the cost of retraining by enabling quick and accurate adaptation for a wide-variety of tasks and models. This is made possible by a combination of weight and function-space priors to reconstruct the gradients of the past, which recovers and generalizes many existing, but seemingly-unrelated, adaptation strategies. Training with simple first-order gradient methods can often recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data. Empirical results show that adaptation with K-priors achieves performance similar to full retraining, but only requires training on a handful of past examples.

Introduction

In production environments, machine learning models are rarely static. Data distributions drift, new information arrives, old data must be removed for privacy (unlearning), and model architectures often need to be swapped for efficiency or compression. Traditionally, these shifts necessitate full retraining, a process that is computationally expensive and environmentally taxing. While specialized techniques like knowledge distillation or weight-priors for continual learning exist, they are often task-specific and lack a unified framework. Knowledge-adaptation priors (K-priors) address this gap by providing a generic mechanism to help a model "pivot" quickly when its environment changes.

Methodology: Reconstructing the Gradients of the Past

The core philosophy of K-priors is that adaptation is gradient reconstruction. If a model can reconstruct the gradient of its original training objective using only a small amount of information, it can update its weights as if it had access to the entire original dataset.

The K-priors framework enables model adaptation by combining past model parameters and a small memory of past data.

K-priors achieve this through several key technical components:

  • Hybrid Divergences: They combine weight-space divergences (keeping parameters close to the previous model) and function-space divergences (ensuring predictions remain similar at specific points).
  • Bregman Divergences: By utilizing the mathematical properties of Bregman divergences, the gradient of the prior is designed to exactly match the gradient of the original loss function.
  • Memorable Past: Rather than storing random examples, K-priors utilize a small "memory" of past data points where the model's predictions are most sensitive (areas of high curvature).
  • Theoretical Unification: K-priors generalize existing methods. For example, removing the weight-space term recovers a form of Knowledge Distillation, while applying the framework to linear settings recovers classic SVM adaptation strategies.

Experimental Evaluation

The authors evaluated K-priors across a diverse set of benchmarks, including tabular data (UCI Adult), handwritten digits (USPS and MNIST), and image classification (CIFAR-10). These datasets were used to test four distinct adaptation tasks: adding new data, removing old data, changing regularizers, and changing model architectures (e.g., compressing a larger network into a smaller one).

Key Results and Insights

K-priors demonstrate significant improvements in both data efficiency and training speed compared to standard "Replay" methods (which simply retrain on a subset of old data using hard labels).

A comprehensive overview of K-priors: (Left) The adaptation framework; (Middle) Visualization of decision boundary adaptation on the Moons dataset; (Right) Validation accuracy across four adaptation tasks.
  • Efficiency: On the USPS digit task, K-priors matched the accuracy of full batch retraining using only 2–5% of the past data.
  • Speedup: For neural networks, K-priors reached 97% accuracy in approximately 54,000 backward passes, compared to 2.7 million for full batch training—a 50x improvement.
  • Soft Labels vs. Hard Labels: The model's previous predictions (soft labels) were found to be more informative than original ground-truth labels for adaptation, as they better characterize the existing decision boundary.
  • Versatility: The framework successfully handled architectural changes, proving that the "memory" of the past can be transferred even when the parameter space changes.

Limitations and Future Work

While highly effective, K-priors require careful tuning of temperature parameters and the balance between weight and function-space priors. In non-linear models like deep neural networks, gradient reconstruction is not mathematically perfect, leaving a small "left-over" term that warrants further study. Future research may explore more advanced "matrix sketching" techniques to further optimize memory selection and test the scalability of K-priors on massive Transformer-scale models.

Summary updated

Abstract

Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch. We present Knowledge-adaptation priors (K-priors) to reduce the cost of retraining by enabling quick and accurate adaptation for a wide-variety of tasks and models. This is made possible by a combination of weight and function-space priors to reconstruct the gradients of the past, which recovers and generalizes many existing, but seemingly-unrelated, adaptation strategies. Training with simple first-order gradient methods can often recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data. Empirical results show that adaptation with K-priors achieves performance similar to full retraining, but only requires training on a handful of past examples.
BibTeX
@inproceedings{khan2021knowledge,
  author = {Mohammad Emtiyaz Khan and Siddharth Swaroop},
  title = {{Knowledge-Adaptation Priors}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=_cXX-Dr7sf0},
}
Figure