Calibrated Learning to Defer with One-vs-All Classifiers

Rajeev Verma, Eric T. Nalisnick

Abstract

The learning to defer (L2D) framework has the potential to make AI systems safer. For a given input, the system can defer the decision to a human if the human is more likely than the model to take the correct action. We study the calibration of L2D systems, investigating if the probabilities they output are sound. We find that Mozannar & Sontag's (2020) multiclass framework is not calibrated with respect to expert correctness. Moreover, it is not even guaranteed to produce valid probabilities due to its parameterization being degenerate for this purpose. We propose an L2D system based on one-vs-all classifiers that is able to produce calibrated probabilities of expert correctness. Furthermore, our loss function is also a consistent surrogate for multiclass L2D, like Mozannar & Sontag's (2020). Our experiments verify that not only is our system calibrated, but this benefit comes at no cost to accuracy. Our model's accuracy is always comparable (and often superior) to Mozannar & Sontag's (2020) model's in tasks ranging from hate speech detection to galaxy classification to diagnosis of skin lesions.

Introduction

In high-stakes domains such as medical diagnosis or autonomous navigation, AI systems are increasingly designed to collaborate with human experts. The Learning to Defer (L2D) framework formalizes this collaboration: an AI model predicts whether it should handle a task itself or "defer" it to a human expert based on who is more likely to be correct. For such a system to be trustworthy and safe, it must be calibrated. A calibrated system's confidence estimates should reflect true likelihoods; for instance, if the system predicts a 70% probability that an expert will be correct, the expert should indeed be correct in approximately 70% of such cases.

The Fragility of Softmax-based L2D

Current state-of-the-art L2D methods, such as the framework proposed by Mozannar & Sontag (2020), typically rely on a Softmax-based multiclass architecture. However, this research reveals a fundamental flaw in that approach. Because the Softmax function couples all class probabilities and the "defer" option into a single distribution, the resulting estimates for expert correctness can become "pathological."

Specifically, the mathematical structure of these models does not constrain the estimated probability of human correctness to the valid [0, 1] range. In empirical tests on the CIFAR-10 dataset, nearly 40% of test samples resulted in an impossible expert correctness probability greater than 1.0. This lack of calibration leads to high Expected Calibration Error (ECE) and makes the system's internal reasoning unreliable for risk assessment.

Pathological probabilities and calibration error in the Softmax-based L2D baseline. Subfigure (a) shows expert correctness probabilities exceeding 1.0, while (b) illustrates a high ECE of 7.58%.

Calibrating L2D with One-vs-All Classifiers

To address these issues, we propose a framework based on One-vs-All (OvA) classifiers. Instead of a single Softmax pool, the problem is decomposed into $K+1$ independent binary classification tasks—one for each of the $K$ classes and one for the "defer" option.

  1. Binary Decomposition: Each task is treated independently, ensuring that the output for each class and the deferral option naturally stays within the valid [0, 1] range.
  2. Consistent Surrogate Loss: We employ a binary surrogate loss (such as logistic loss) for each task. We prove that this OvA loss is a "consistent surrogate," meaning that with sufficient data, the model is guaranteed to converge to the optimal decision-making strategy.
  3. ECOC Integration: By utilizing Error Correcting Output Codes (ECOC), we provide a theoretical foundation for the consistency of our multiclass L2D surrogate.

Experimental Results

The OvA framework was evaluated across four diverse datasets: CIFAR-10 (image recognition), HateSpeech (text detection), Galaxy-Zoo (astronomy), and HAM10000 (medical dermatology).

Calibration and Accuracy

The results demonstrate that calibration does not come at the cost of performance. On CIFAR-10, the OvA method reduced the ECE from 7.58% to 3.01% while maintaining or improving overall system accuracy. In the medical context of the HAM10000 dataset, the OvA model significantly improved the faithfulness of error predictions, reducing the gap between predicted and true error by a factor of three.

Comparison of system and classifier accuracy between Softmax and OvA frameworks. OvA (blue) consistently matches or outperforms the Softmax baseline (red) across different levels of expert expertise (k) and coverage.

As shown in the figures above, the OvA model (blue) achieves higher system accuracy than the Softmax baseline (red), particularly as the "budget" for human intervention (coverage) varies. This suggests that a well-calibrated model enables a more efficient division of labor between the AI and the human expert.

Conclusion

The proposed One-vs-All framework for Learning to Defer fixes the "range problem" inherent in Softmax-based systems. By producing valid, calibrated probabilities of expert correctness, it enhances the safety and interpretability of human-AI collaboration. Crucially, these gains in reliability are achieved without sacrificing—and often while improving—the predictive accuracy of the overall system.

Summary updated

Abstract

The learning to defer (L2D) framework has the potential to make AI systems safer. For a given input, the system can defer the decision to a human if the human is more likely than the model to take the correct action. We study the calibration of L2D systems, investigating if the probabilities they output are sound. We find that Mozannar & Sontag's (2020) multiclass framework is not calibrated with respect to expert correctness. Moreover, it is not even guaranteed to produce valid probabilities due to its parameterization being degenerate for this purpose. We propose an L2D system based on one-vs-all classifiers that is able to produce calibrated probabilities of expert correctness. Furthermore, our loss function is also a consistent surrogate for multiclass L2D, like Mozannar & Sontag's (2020). Our experiments verify that not only is our system calibrated, but this benefit comes at no cost to accuracy. Our model's accuracy is always comparable (and often superior) to Mozannar & Sontag's (2020) model's in tasks ranging from hate speech detection to galaxy classification to diagnosis of skin lesions.
BibTeX
@inproceedings{verma2022calibrated,
  author = {Rajeev Verma and Eric T. Nalisnick},
  title = {{Calibrated Learning to Defer with One-vs-All Classifiers}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=acOfwl5NIE},
}
Figure