Abstract: Calibration is a critical property for establishing the trustworthiness of predictors that provide uncertainty estimates. Multicalibration is a strengthening of calibration which requires that predictors be calibrated on a potentially overlapping collection of subsets of the domain. As multicalibration grows in popularity with practitioners, an essential question is: how do we measure how multicalibrated a predictor is? Błasiok et al. (2023) considered this question for standard calibration by introducing the distance to calibration framework (dCE) to understand how calibration metrics relate to each other and the ground truth. Building on the dCE framework, we consider the auditability of the distance to multicalibration of a predictor $f$. We begin by considering two natural generalizations of dCE to multiple subgroups: worst group dCE (wdMC), and distance to multicalibration (dMC). We argue that there are two essential properties of any multicalibration error metric: 1) the metric should capture how much $f$ would need to be modified in order to be perfectly multicalibrated; and 2) the metric should be auditable in an information theoretic sense. We show that wdMC and dMC each fail to satisfy one of these two properties, and that similar barriers arise when considering the auditability of general distance to multigroup fairness notions. We then propose two (equivalent) multicalibration metrics which do satisfy these requirements: 1) a continuized variant of dMC; and 2) a distance to intersection multicalibration, which leans on intersectional fairness desiderata. Along the way, we shed light on the loss-landscape of distance to multicalibration and the geometry of the set of perfectly multicalibrated predictors. Our findings may have implications for the development of stronger multicalibration algorithms as well as multigroup auditing more generally.
Introduction
In high-stakes machine learning applications, such as medical diagnostics or financial risk assessment, it is insufficient for a model to be merely accurate; it must also be calibrated. A model is calibrated if its predicted probability of an event matches the observed frequency of that event. Multicalibration extends this requirement to a collection of potentially overlapping subgroups (e.g., age, race, and geography), ensuring that the model's uncertainty estimates are reliable for all protected or relevant populations simultaneously.
As multicalibration becomes a standard for algorithmic fairness, a fundamental question arises: how should we measure the "distance" between a given predictor $f$ and the set of perfectly multicalibrated predictors? This paper formalizes the requirements for such a metric and evaluates existing candidates against these desiderata.
Desiderata for Multicalibration Metrics
The authors identify two essential properties that any robust multicalibration metric must satisfy:
- Optimization-Friendly Landscape: The metric's loss landscape should be well-behaved. Specifically, it should avoid "fairness traps"—local minima that are not global minima. If a metric has bad local minima, an optimization algorithm might conclude that a model cannot be made fairer, even when a perfectly multicalibrated model exists nearby.
- Auditability: The metric must be estimable from a finite number of data samples. Mathematically, this requires the metric to be continuous (specifically Lipschitz continuous) with respect to the underlying data distribution. If a microscopic change in the ground truth causes a massive jump in the metric, it is considered "inauditable" because it cannot be reliably measured from empirical data.
The Geometry of Multicalibration Landscapes
We evaluate two natural generalizations of the distance to calibration (dCE) framework: Worst-group distance to Multicalibration (wdMC) and Distance to Multicalibration (dMC).
Analysis reveals a fundamental trade-off. While wdMC is intuitive—measuring the distance to calibration for the worst-performing subgroup—it fails the landscape requirement. As shown in Figure 1, the wdMC landscape contains local minima that are not global minima. In contrast, the dMC metric, which measures the distance to the nearest set of perfectly multicalibrated predictors, possesses a "nice" landscape where local minima are global minima.
The Auditability Barrier
Although dMC has a superior optimization landscape, it fails the second desideratum: auditability. The authors prove that dMC is discontinuous with respect to the ground truth distribution. A tiny shift in the distribution can cause the set of perfectly multicalibrated predictors to change abruptly, leading to a jump in the distance metric.
This inauditability means that dMC cannot be estimated with any finite number of samples without making strong assumptions about the simplicity of the ground truth. This finding highlights a significant challenge for practitioners attempting to audit multigroup fairness notions using standard distance-based approaches.
Proposed Solution: Continuization and Intersectionality
To resolve this tension, the authors propose a "continuized" variant of dMC. By smoothing out the mathematical discontinuities, they derive a metric that is both auditable and maintains a clean optimization landscape.
Remarkably, they prove that this continuized metric is mathematically equivalent to Distance to Intersection Multicalibration (dIMC). This metric requires calibration on all possible intersections of the defined subgroups. This result provides a theoretical bridge between multicalibration and intersectional fairness, suggesting that intersectionality is not just a social desideratum but a mathematical necessity for auditable fairness metrics.
Conclusion and Future Work
This work establishes a formal hierarchy of multicalibration metrics and identifies dIMC as the most viable candidate for both optimization and auditing. However, challenges remain, particularly the "curse of dimensionality": auditing intersectional fairness can require a sample size that grows exponentially with the number of subgroups. Future research may explore how assuming "simple" ground truths (e.g., low VC dimension) could mitigate these statistical costs, potentially leading to more efficient auditing procedures for complex, real-world datasets.
Summary updated