Weight-Based Performance Estimation for Diverse Domains

Ekaterina Khramtsova, Mahsa Baktashmotlagh, Guido Zuccon, Xi Wang, Mathieu SalzmannOpenReview 2024 · Submission to ICLR 2024 · Rejected · OpenReview

Abstract

One of the limitations of applying machine learning methods in real-world scenarios is the existence of a domain shift between the source (i.e., training) and target (i.e., test) datasets, which typically entails a significant performance drop. This is further complicated by the lack of annotated data in the target domain, making it impossible to quantitatively assess the model performance. As such, there is a pressing need for methods able to estimate a model's performance on unlabeled target data. Most of the existing approaches addressing this train a linear performance predictor, taking as input either an activation-based or a performance-based metric. As we will show, however, the accuracy of such predictors strongly depends on the domain shift. By contrast, we propose to use a weight-based metric as input to the linear predictor. Specifically, we measure the difference between the model's weights before and after fine-tuning it on a self-supervised loss, which we take to be the entropy of the network's predictions. This builds on the intuition that target data close to the source domain will produce more confident predictions, thus leading to small weight changes during fine-tuning. Our extensive experiments on standard object recognition benchmarks, using diverse network architectures, demonstrate the benefits of our method, outperforming both activation-based and performance-based baselines by a large margin. Our code is available in an anonymous repository: https://anonymous.4open.science/r/79E9/
BibTeX
@misc{khramtsova2024weight,
  author = {Ekaterina Khramtsova and Mahsa Baktashmotlagh and Guido Zuccon and Xi Wang and Mathieu Salzmann},
  title = {{Weight-Based Performance Estimation for Diverse Domains}},
  year = {2024},
  url = {https://openreview.net/forum?id=OHll7EfuSi},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2024},
}
Figure