Abstract
Traditional AI trustworthiness evaluations prioritize model calibration—the alignment of predicted probabilities with observed frequencies. However, for decision-making tasks, calibration can be a misleading metric. This paper, presented at AAAI-24, introduces $\mathcal{U}$-trustworthiness, a framework centered on utility maximization. We demonstrate that Area Under the Curve (AUC) serves as a superior metric for identifying models capable of optimal decision-making, showing that properly-ranked models can achieve optimal utility regardless of their probabilistic alignment.
Introduction: The Calibration Trap
In the current AI landscape, a "trustworthy" model is usually defined as one that is fair, robust, and well-calibrated. Calibration is often treated as a prerequisite: if a model predicts a 10% risk of loan default and the actual frequency is 10%, we "trust" it. However, this narrow view ignores the ultimate goal of most AI systems: making high-utility decisions.
A model can be perfectly calibrated but still lead to poor decisions if it lacks the "competence" to distinguish between high-utility and low-utility outcomes. Conversely, a model might provide "incorrect" probabilities but maintain "correct" rankings, allowing a user to select a decision threshold that maximizes benefit. This paper shifts the focus from probabilistic accuracy to decision-making competence.
The $\mathcal{U}$-Trustworthiness Framework
Grounding their work in philosophical competence-based trust theory, the authors define a model as $\mathcal{U}$-trustworthy if it is capable of achieving the maximum possible utility for a specific class of tasks. This framework rests on three pillars:
- Reliance: The model provides a decision boundary.
- Competence: No other model can achieve higher utility for the task.
- Confidence: There is statistical evidence supporting the model's competence.
A key contribution is the concept of a properly-ranked classifier. Such a model preserves the same relative ordering of instances as a theoretical "Bayes optimal" classifier, even if its raw scores are not well-calibrated probabilities.
AUC as the Measure of Competence
The authors provide mathematical proofs showing that for a wide range of utility functions—including those that are cost-sensitive or equity-aware—a properly-ranked model is inherently competent. Crucially, they prove that if a model’s AUC matches the Bayes optimal AUC, it is guaranteed to be $\mathcal{U}$-trustworthy.
This elevates AUC from a simple performance metric to a formal guarantee of decision-making competence. For decision-making, the relative order of samples is more important than the absolute probability score. If the ranking is correct, the "reliability" comes from choosing the right decision threshold to maximize utility.
Empirical Validation
The theory was validated using both synthetic simulations and the American Housing Survey (2019) dataset.
- Synthetic Tests: In 400 datasets, the authors compared Bayes optimal, properly-ranked, and calibrated-adversarial models. They found that miscalibrated models often achieved the same peak utility as the Bayes optimal, while perfectly calibrated models with poor rankings fell short.
- Real-World Application: On the housing dataset, Logistic Regression appeared more "trustworthy" via traditional metrics like the Brier score. However, Random Forest—which had a higher AUC—consistently achieved higher maximum utility across various cost-benefit scenarios.
Limitations and Future Work
While the framework provides a robust foundation for binary classification, extending it to multi-class problems remains a future challenge. Additionally, the authors note that performing hypothesis testing on AUC values to prove "Bayes-level" competence (the "Confidence" pillar) is a difficult open research problem. Finally, the framework assumes utility maximization as the primary goal; it may not apply to fields like insurance or weather forecasting where raw probability scores are the required output for risk pooling.
Conclusion
The findings suggest that model selection and hyperparameter tuning should prioritize AUC over accuracy or calibration when the goal is decision-making. Trustworthiness is not a static property of a model but a relationship between the model and a specific utility goal. By focusing on ranking competence rather than probabilistic alignment, practitioners can build AI systems that are more effective in real-world applications.
Summary updated