The Hitchhiker's Guide to Prior-Shift Adaptation

Tomáš Šipka, Milan Šulc, Jiří MatasWACV 2022

TL;DR

This paper tackles the challenge of adapting machine learning classifiers when the distribution of class labels in the test data differs from the training data (prior shift). They propose a novel, robust method for estimating these new priors and demonstrate its effectiveness, showing it outperforms existing techniques and leads to significant accuracy improvements in real-world scenarios.

What Problem Does This Paper Address?

In many real-world applications, machine learning models are trained on data with a certain distribution of class labels. However, when deployed, the actual distribution of these labels in the test data can change. This phenomenon, known as "prior shift" or "label shift," can significantly degrade a classifier's performance because its predictions are based on the assumption of the original training distribution. For instance, a medical diagnostic model trained on data with a low prevalence of a certain disease might perform poorly during an outbreak when that disease becomes much more common. The paper aims to develop methods that can adapt a pre-trained classifier to these new, unknown class priors without requiring retraining, or by efficiently retraining with adjusted data sampling.

Methods

The authors propose a novel method called SCML (Soft Confusion Matrix Likelihood) for estimating new class priors from unlabeled test data. This method addresses a key issue in existing approaches that rely on confusion matrices: inconsistent estimates can lead to nonsensical negative prior probabilities. SCML works by maximizing the likelihood of classifier decisions, treating the confusion matrix and test data distribution as potentially noisy estimates. They also extend this to a Maximum A Posteriori (MAP) estimator (SCMM) by incorporating a hyper-prior, which further improves performance, especially for dense class distributions.

The paper also provides a comprehensive review and experimental comparison of existing prior shift adaptation techniques, categorizing them into:

  • Adapting predictions of a fixed classifier: This involves re-weighting the classifier's output probabilities based on the estimated new priors.
  • Re-training the classifier: This involves adjusting the training data sampling strategy to match the new priors.

They also investigate the role of classifier calibration (e.g., temperature scaling) in prior shift adaptation, finding that while it can be beneficial, certain methods like BCTS might not be universally suitable, especially for long-tailed distributions.

Datasets

The paper uses several standard long-tailed datasets for evaluation:

  • CIFAR100-LT*: A long-tailed version of CIFAR100.
  • Places365-LT: A long-tailed version of Places365.
  • ImageNet-LT: A long-tailed version of ImageNet.

They also evaluate their methods on practical tasks:

  • PlantCLEF: A fine-grained plant classification dataset.
  • Webvision 1.0: A noisy, long-tailed dataset for learning ImageNet classes.

Additionally, they use uniformly distributed test sets (UNI) to evaluate adaptation from long-tailed training sets.

Key Insights and Ideas

  • Direct adaptation is often better than retraining: Adapting the predictions of an already trained classifier is generally more effective and computationally cheaper than retraining the entire model with adjusted sampling strategies.
  • Robust prior estimation is crucial: Existing methods based on confusion matrix inversion can produce invalid (negative) prior estimates. The proposed SCML method effectively handles these inconsistencies.
  • Soft confusion matrices improve performance: Using soft predictions from the classifier to estimate the confusion matrix (SCM) consistently yields better results than using hard predictions.
  • MAP estimation offers further gains: The SCMM method, incorporating a hyper-prior, provides an additional boost in performance, particularly for dense class distributions.
  • Calibration matters, but not always: While classifier calibration can be beneficial, it's not a silver bullet. Some calibration methods might even degrade performance on long-tailed datasets.
  • Prior estimation requires sufficient data: The accuracy of prior estimation methods is dependent on the number of test samples. With very few samples, it might be better to omit prior estimation altogether.

Key Results

  • The proposed SCML method consistently outperforms existing confusion matrix-based prior estimation methods, achieving the best results when using soft confusion matrices.
  • In most scenarios, adaptation of a pre-trained classifier using SCML or SCMM achieved higher accuracy than retraining the classifier with sampling matching the new priors.
  • On practical tasks, applying the best practices (calibration, SCML estimation, and adaptation) led to significant accuracy improvements: 1.1% on web-crawled images and 3.4% on plant species classification.
  • The EM algorithm for prior estimation performed better than SCML with a very small number of test samples.

Limitations and Future Work

  • Dependence on test sample size: The accuracy of prior estimation methods is sensitive to the number of test samples available. The paper suggests omitting estimation with extremely few samples.
  • Calibration effectiveness: The paper notes that certain calibration methods (like BCTS) might not be reliable for adaptation on long-tailed datasets, suggesting further research into calibration strategies for such scenarios.
  • Generalizability of BCTS: The paper indicates that BCTS might overfit to validation set errors when class-specific parameters are used, suggesting a need for more robust calibration methods.

Future work could explore more advanced calibration techniques tailored for long-tailed distributions and investigate the impact of different hyper-priors in the MAP estimation framework.

Summary updated

BibTeX
@inproceedings{sipka2022hitchhiker,
  author = {Tomáš Šipka and Milan Šulc and Jiří Matas},
  title = {{The Hitchhiker's Guide to Prior-Shift Adaptation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure