Knowledge-Adaptation Priors

Mohammad Emtiyaz Khan, Siddharth Swaroop

Overview

In production environments, machine learning models are rarely static. Data drifts, new labels arrive, old data becomes irrelevant, and sometimes even the model architecture needs to change (e.g., for model compression). Currently, the standard solution to these shifts is to retrain the model from scratch on the entire accumulated dataset. While effective, this approach is financially and environmentally expensive.

Knowledge-Adaptation Priors (K-priors) offer a highly efficient alternative. They allow machine learning models to adapt to changes using only a tiny fraction (2-10%) of past data. By combining weight-space and function-space divergences, K-priors reconstruct the gradients of the past, matching the performance of full retraining at a fraction of the computational cost.

The Challenge of Model Adaptation

When a deployed model encounters new data or requires an architectural update, engineers need a way to integrate the new information without losing the valuable patterns learned from the past. Existing "quick" fixes often fall short:

  • Weight-priors: Common in continual learning, these methods penalize changes to the model's weights. However, they often fail when the task or architecture changes significantly, as the old weights may no longer be relevant to the new structure.
  • Replay methods: These involve storing a subset of old data and mixing it with new data during training. Unfortunately, replay requires large memory banks to be effective and often struggles with "catastrophic forgetting" when memory is constrained.

The authors seek a generic adaptation mechanism that works across different models (from Generalized Linear Models to Deep Networks) and various adaptation tasks without requiring the full history of data.

Knowledge-Adaptation Priors (K-Priors)

The core innovation of this paper is the K-prior, a specialized prior that regularizes both the model parameters (weights) and the model's predictions (function space).

The K-prior is defined mathematically as: $$K(w; w_*, \mathcal{M}) = \mathbb{D}_f(\mathbf{f}(w) || \mathbf{f}(w_*)) + \tau \mathbb{D}_w(w || w_*)$$

Where:

  • $\mathbb{D}_f$ is a function-space divergence that ensures the new model's predictions on a small memory set $\mathcal{M}$ match the old model's predictions ($w_*$).
  • $\mathbb{D}_w$ is a weight-space divergence that keeps the new parameters close to the old ones.
  • $\mathcal{M}$ is a small "memory" of past examples.

The "secret sauce" of this formulation is that it is designed to reconstruct the gradient of the past training objective. For Generalized Linear Models (GLMs), the authors prove that K-priors can exactly recover the retrained model's gradient. For deep networks, it provides a powerful approximation.

To keep the memory $\mathcal{M}$ small, the authors employ a "memorable past" strategy: selecting examples where the model's predictions are most sensitive (highest derivative of the loss), rather than just picking points at random.

Overview of K-priors: adapting models to new tasks using a small memory of past data, matching full retraining performance.

Core Mechanisms and Insights

  • Gradients are the Key: Instead of trying to match the old loss, K-priors aim to match the old gradient. This allows the optimizer to "feel" the influence of the entire past dataset while only looking at a few representative examples.
  • Soft Labels > Hard Labels: K-priors use the old model's predictions (soft labels) on the memory set. This provides much richer information about the decision boundary than using the original ground-truth labels (hard labels), which is the standard practice in Replay methods.
  • Unification of Techniques: The paper demonstrates that K-priors provide a theoretical bridge between seemingly unrelated fields: Knowledge Distillation (which acts as a K-prior without the weight term), Support Vector Machines (SVMs), and Gaussian Processes.
  • Data Efficiency: For many tasks, a massive archive is unnecessary. Storing just 2-5% of the most "memorable" past examples is often enough to stay within 1% of the accuracy of a model retrained on 100% of the data.

Empirical Results

The authors evaluated K-priors on several standard benchmarks, including UCI Adult, USPS, MNIST, and CIFAR-10. The results demonstrate significant improvements over baselines across multiple adaptation scenarios:

  • Matching the "Batch" Gold Standard: In experiments on the Adult and USPS datasets, K-priors consistently matched the performance of full batch retraining while using only a tiny fraction of past data.
  • Outperforming Replay: Across all tested tasks—adding data, removing data, changing regularizers, and changing architectures—K-priors significantly outperformed standard Replay, especially in highly memory-constrained regimes.
  • Architecture Adaptation: K-priors successfully adapted models when the architecture changed (e.g., compressing a 2-hidden-layer MLP to a 1-hidden-layer MLP), a scenario where traditional weight-priors typically fail.
  • Deep Learning Potential: On CIFAR-10, while a small gap remained between K-priors and full retraining, K-priors still comfortably outperformed Replay. Performance improved further when using a "temperature" parameter to soften predictions, akin to Knowledge Distillation.

Limitations and Future Directions

While highly effective, K-priors do have limitations. For complex deep learning tasks, they require careful tuning of hyperparameters, such as the temperature and the trade-off between weight and functional regularization ($\tau$). Furthermore, while the theory is general, empirical results on large-scale vision tasks show a slight performance gap compared to full retraining that future work will need to close. Finally, while the "memorable past" heuristic works well, the authors suggest that more sophisticated "sketching" methods (like leverage scores) could further optimize memory selection and accuracy.

Summary updated

Abstract

Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch. We present Knowledge-adaptation priors (K-priors) to reduce the cost of retraining by enabling quick and accurate adaptation for a wide-variety of tasks and models. This is made possible by a combination of weight and function-space priors to reconstruct the gradients of the past, which recovers and generalizes many existing, but seemingly-unrelated, adaptation strategies. Training with simple first-order gradient methods can often recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data. Empirical results show that adaptation with K-priors achieves performance similar to full retraining, but only requires training on a handful of past examples.
BibTeX
@inproceedings{khan2021knowledge,
  author = {Mohammad Emtiyaz Khan and Siddharth Swaroop},
  title = {{Knowledge-Adaptation Priors}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=_cXX-Dr7sf0},
}

Loading related content...

Figure