KULeuven at LeQua 2022: Model Calibration in Quantification Learning

Teodora Popordanoska, Matthew B. BlaschkoCLEF (Working Notes) 2022 · OpenReview

Abstract

Quantification learning focuses on estimating class prevalences in populations under distribution shift. This paper examines the relationship between model calibration and quantification bias. We demonstrate that minimizing calibration error is an effective proxy for reducing bias. Through experiments on the LeQua 2022 vector task, we show that simple post-hoc Platt scaling combined with the Saerens-Latinne-Decaestecker (SLD) algorithm outperforms complex trainable calibration methods. Our best-performing model, a calibrated Support Vector Machine, secured 1st place in the competition for relative absolute error.

Introduction

In many real-world applications, the primary interest lies in the prevalence of classes across a population rather than individual predictions. This task, known as quantification learning, is complicated by distribution shifts where training and test class proportions differ significantly. Standard classifiers often yield biased prevalence estimates in these scenarios. This work investigates the theoretical link where $L_1$ calibration error provides an upper bound on quantification bias, exploring whether explicit calibration can improve robustness to shift.

Methods

We evaluate two primary calibration strategies:

  1. Post-hoc Calibration (Platt Scaling): A standard technique that applies a sigmoid transformation to model outputs after training.
  2. Trainable Calibration (KDE-XE): An approach that incorporates Kernel Density Estimation into the loss function during training to minimize calibration error directly.

These calibrated models serve as inputs for the Saerens-Latinne-Decaestecker (SLD) algorithm, which uses an expectation-maximization framework to iteratively refine prevalence estimates.

Experimental Setup

Experiments were conducted using the LeQua 2022 (Task T1A) dataset, consisting of vector representations of Amazon product reviews. The training set comprised 5,000 instances. Test samples were generated via the Artificial Prevalence Protocol (APP), which artificially manipulates class distributions to create a uniform distribution of prevalences, thereby testing the models' robustness to extreme shifts.

Results

Our results indicate that simplicity often prevails in calibration for quantification. Despite the theoretical appeal of trainable calibration, post-hoc Platt scaling consistently yielded better results. Notably, a Support Vector Machine (SVM) with an RBF kernel, calibrated via Platt scaling, outperformed Multi-Layer Perceptrons (MLPs) and Logistic Regression.

Performance comparison of various calibration methods and classifiers on the LeQua 2022 dataset.

As shown in the table above, the SLD_PS_SVM configuration achieved the best performance with an MRAE of 0.117 and an accuracy of 0.881. In contrast, the KDE-XE method (e.g., SLD_KDE_LR) struggled with the severe shifts present in the APP protocol, resulting in significantly higher error rates compared to its Platt-scaled counterparts.

Conclusion

This study confirms that minimizing calibration error is a vital step for accurate quantification. However, the choice of calibration method is crucial; post-hoc methods like Platt scaling proved more resilient to the severe prevalence shifts found in the LeQua competition than more complex trainable alternatives. Future research will focus on characterizing the types of distribution shifts where different calibration strategies excel and extending these methods to multi-class problems.

Summary updated

BibTeX
@inproceedings{popordanoska2022kuleuven,
  author = {Teodora Popordanoska and Matthew B. Blaschko},
  title = {{KULeuven at LeQua 2022: Model Calibration in Quantification Learning}},
  booktitle = {CLEF (Working Notes)},
  year = {2022},
  url = {https://openreview.net/forum?id=ZjkFDHfMGbG},
}
Figure