Abstract
Traditional evaluation of probabilistic classifiers relies on global metrics like accuracy and Expected Calibration Error (ECE). However, these metrics can mask significant local biases that undermine a model's reliability for specific inference tasks. This paper introduces the I-trustworthy framework, which formalizes trustworthiness through the lens of local calibration. We present a novel kernel-based statistical test (KLCE) and diagnostic tools to identify and localize hidden biases that standard metrics fail to detect.
Introduction
In high-stakes applications such as healthcare and criminal justice, the "average" performance of a machine learning model is an insufficient measure of its reliability. A model may exhibit high global accuracy while consistently producing biased predictions for specific demographic subgroups. Such "local" failures can lead to skewed results in downstream inference tasks, such as estimating population-level statistics from individual risk scores. To address this, we propose the I-trustworthy framework, asserting that a model is only trustworthy for a given task if it maintains local calibration across the relevant feature space.
The I-trustworthy Framework and KLCE
The core requirement of the I-trustworthy framework is local calibration: for any specific combination of input features, the model's predicted probability must coincide with the true conditional probability of the outcome. This is a more rigorous standard than global calibration, which only requires alignment on average across the entire dataset.
To test for local calibration, we introduce the Kernel Local Calibration Error (KLCE). By leveraging Reproducing Kernel Hilbert Spaces (RKHS), KLCE measures discrepancies between predictions and outcomes across a secondary feature space of interest (e.g., protected attributes like race or age). This formulation allows for a formal hypothesis testing procedure: if the KLCE is significantly greater than zero, the model's claim of trustworthiness for that task can be statistically rejected.
Empirical Validation: The Accuracy Paradox
We evaluated the framework using several real-world datasets, including the American Housing Survey (AHS) and the COMPAS recidivism dataset. Our results reveal a critical "accuracy paradox": models that appear superior under standard metrics can be fundamentally untrustworthy for specific applications.
As illustrated in the figure above, Model 1 exhibits a lower Brier Score and higher Accuracy than Model 2. However, when these models are used to infer the average age of homeowners, Model 1 produces a significantly biased estimate. In contrast, Model 2—despite its lower global performance—provides an estimate centered on the true value. This demonstrates that global metrics are poor proxies for trust in specific inference contexts.
Model Criticism and Diagnosis
Beyond detection, the framework provides a diagnostic tool called Local Calibration Bias (LCB). LCB acts as an "error witness function," allowing practitioners to map exactly where a model fails. In our audit of the COMPAS dataset, the KLCE test successfully flagged models as untrustworthy (p < 0.002) even after the application of standard recalibration techniques like Platt scaling. The LCB tool pinpointed specific demographic slices where systemic misprediction persisted, providing a clear path for model refinement.
Conclusion
The I-trustworthy framework provides a principled approach to auditing probabilistic classifiers. By shifting the focus from global averages to local calibration, we can detect and diagnose biases that threaten the integrity of AI-driven decision-making. While challenges remain—such as the sensitivity of kernel methods to high-dimensional data—this framework represents a vital step toward developing truly accountable and trustworthy machine learning systems.
Summary updated