A Bayesian Divergence Prior for Classiffier Adaptation

Xiao Li, Jeff A. BilmesAISTATS 2007

Abstract

Model adaptation is a critical challenge when deploying machine learning systems in environments where target data distributions shift from training distributions. This paper introduces a Bayesian "divergence prior" that unifies adaptation strategies for both generative and discriminative models. By mathematically balancing the fit on new data with proximity to the original model, the proposed framework provides a principled alternative to ad-hoc adaptation techniques. We provide PAC-Bayesian theoretical guarantees showing that this approach yields tighter generalization bounds than standard training when the target distribution is related to the source. Empirical evaluations on speech and vision datasets demonstrate significant performance gains, particularly in low-data regimes.

Introduction

Standard machine learning assumes that training and testing data are drawn from the same underlying distribution. However, real-world applications often face distribution shifts—for instance, a speech recognition model trained on diverse voices must adapt to a specific user's vocal characteristics. This creates an adaptation problem: given a pre-trained "unadapted" classifier and a limited amount of labeled "adaptation data" from a target distribution, how can we refine the model without overfitting?

A significant constraint in many scenarios is the lack of access to the original training data due to privacy, storage, or computational limits. Simple retraining on small adaptation datasets typically leads to massive overfitting, while failing to adapt results in suboptimal performance. This work seeks a mathematically rigorous way to adapt models using a divergence prior that penalizes deviations from the original model.

Methodology

The authors propose an "accuracy-regularization" approach within a Bayesian framework. Instead of an uninformative prior, they utilize a divergence prior defined after the unadapted model is trained. The objective function minimizes empirical risk on the new dataset while maximizing the divergence prior, which acts as a penalty forcing the new model to remain "close" to the unadapted model as measured by KL-divergence.

Adaptation for Generative and Discriminative Models

For generative models, the objective reduces to minimizing the error on new data plus the KL-divergence between the joint distributions of the old and new models. For discriminative models (e.g., SVMs, MLPs), which only learn the conditional distribution $p(y|x)$, the authors derive a clever upper bound on the divergence that does not require the original data distribution. This derivation justifies common practical regularizers, such as penalizing the $L_2$ distance between new and old weights in neural networks or reusing support vectors in SVMs.

Theoretical Guarantees

Applying PAC-Bayesian theory, the authors prove that using this divergence prior guarantees tighter generalization bounds on target data, provided the target distribution is not excessively distant from the source. The analysis reveals a threshold ($\beta$); if the KL-divergence between source and target distributions is below $\beta$, the adapted model is guaranteed to generalize better than a model trained from scratch.

Experimental Results

The proposed "Regularized" adaptation methods were evaluated on vowel classification (speech) and object recognition (vision) datasets.

Vowel Classification (MLP)

In the speaker adaptation task using Multi-Layer Perceptrons (MLPs), the regularized approach consistently outperformed baselines. With 800 adaptation samples, the regularized MLP achieved an 11.56% error rate, significantly better than the unadapted model (32.03%) and the model retrained from scratch (14.21%).

Table 1: Error rates (%) for MLP adaptation on the Vowel dataset across different sample sizes.

Object Recognition (SVM)

The method's effectiveness in extreme low-data regimes was demonstrated using Support Vector Machines (SVMs) on an object recognition task. With only 90 adaptation samples, the regularized SVM achieved an 11.0% error rate, vastly outperforming the retrained model (30.1%).

Table 2: Error rates (%) for SVM adaptation on the Object dataset.

Discussion and Conclusion

The Bayesian divergence prior provides a unified theoretical foundation for various empirical adaptation tricks. By bounding KL divergence using parameter distance, the authors offer a rigorous justification for weight-proximity regularization in discriminative models.

While the framework is robust, some limitations remain. Exact KL-divergence is not always computable in closed-form for complex models like Gaussian Mixture Models (GMMs), requiring upper-bound approximations. Additionally, the derivations for discriminative classifiers assume specific logistic forms for conditional probabilities. Future work may extend these theoretical results to more complex architectures and provide stricter bounds for continuous models.

Summary updated

BibTeX
@article{li2007bayesian,
  author = {Xiao Li and Jeff A. Bilmes},
  title = {{A Bayesian Divergence Prior for Classiffier Adaptation}},
  journal = {AISTATS},
  year = {2007},
}
Figure