TL;DR
When choosing between machine learning models for downstream decision-making, traditional calibration metrics (like ECE) fail to tell you which model will yield higher payoffs—especially when both models are imperfect. This paper introduces the informativeness gap, an operational metric that quantifies the maximum utility advantage one predictor provides over another across all decision tasks. The authors show this gap can be characterized via a relaxed Earth Mover’s Distance (Wasserstein distance), can be estimated sample-efficiently, and reveals a fundamental impossibility: no informativeness metric can be simultaneously complete, sound, and continuous.
What Problem Does This Paper Address?
Downstream decision-makers often have to choose between competing predictors (e.g., different LLMs or forecasting models), but these models are frequently miscalibrated. Existing calibration metrics like Expected Calibration Error (ECE) simply measure deviation from perfection. Even recent decision-theoretic metrics—such as U-Calibration or Calibration Decision Loss—only assess how much better a single model would be compared to its own calibrated version or a baseline.
Meanwhile, classical decision theory (Blackwell informativeness) only applies to perfectly calibrated models and only provides a partial order: many models simply cannot be compared. This leaves a major practical gap: If you have two distinct, possibly miscalibrated predictors, how can you determine which one will yield higher payoffs in downstream decision tasks, and by how much?
Methods
The authors introduce a framework focusing on binary outcomes:
- Informativeness Gap ($\text{INFOGAP}[\mu, \nu]$): Defined as the worst-case payoff advantage that predictor $\mu$ provides over predictor $\nu$ across all payoff-normalized decision tasks, where an agent naively best-responds to each predictor's probabilistic forecasts.
-
Relaxed Earth Mover's Distance ($\text{REMD}$): For perfectly calibrated predictors with prediction distributions $f_\mu$ and $f_\nu$, the authors prove the gap equals a relaxed variant of the Earth Mover’s (Wasserstein-1) Distance: $$\text{REMD}[f_1, f_2] \triangleq \inf_{\pi \in \Pi(f_1, f_2)} \int_0^1 \left| \int_0^1 \pi(p, q) \cdot (p - q) \, dq \right| dp$$ where $\Pi(f_1, f_2)$ is the set of valid couplings between the two distributions.
-
Miscalibrated Generalization ($\text{REMD}^{\text{MISC}}$): When predictors are miscalibrated, the metric incorporates conditional empirical frequencies $\kappa(p) = \mathbb{P}(\text{outcome}=1 \mid \text{prediction}=p)$ and relaxes the coupling constraints to a broader "flow set" with flow-conservation constraints.
- Prediction-Only Estimation: The authors demonstrate that $\text{REMD}^{\text{MISC}}$ can be estimated directly from observable sample pairs of $(\text{prediction}, \text{realized outcome})$ without needing internal access to model parameters.
Datasets
Not discussed in the paper. This is a purely theoretical paper, though the authors analyze concrete analytical toy examples (such as a 50% base-rate rain forecasting scenario) to build intuition.
Key Insights and Ideas
- Miscalibrated yet highly informative predictors can strictly beat perfectly calibrated ones: A predictor that miscalibrates slightly around 50% but predicts 0% or 100% with near-perfect accuracy will strictly outperform a perfectly calibrated predictor that uninformatively predicts the 50% base rate for every instance.
- The location of miscalibration matters immensely: Two predictors can have identical ECE magnitudes, yet one provides superior decision utility while the other leads to catastrophic decisions. Specifically, miscalibrating on extreme probabilities (e.g., predicting 99% when true rate is 100%) damages decision-making far more severely than miscalibrating around the middle.
- Strict generalization of foundational theory: $\text{INFOGAP}$ generalizes Blackwell’s theorem (1951, 1953) to all pairs of calibrated predictors, while also subsuming U-Calibration (Kleinberg et al., 2023) and Calibration Decision Loss (Hu and Wu, 2024).
- The Consistency vs. Continuity Impossibility: The paper proves an inherent trilemma: no informativeness measure can simultaneously be complete (always ranking if one is weakly better), sound (never falsely claiming dominance), and continuous with respect to the prediction distribution.
Key Results
- Exact Dual Characterization: For calibrated predictors, $\text{INFOGAP}[\mu, \nu] = \text{REMD}[f_\mu, f_\nu]$. For general miscalibrated predictors, $\text{INFOGAP}[\mu, \nu] = \text{REMD}^{\text{MISC}}[f_\mu, f_\nu, \kappa_\mu, \kappa_\nu]$.
- Equivalence at the Boundary: When predictors are perfectly calibrated ($\kappa(p) = p$), the generalized flow-based $\text{REMD}^{\text{MISC}}$ precisely collapses to the coupling-based $\text{REMD}$.
- Efficient Sample Complexity: The authors provide an algorithm showing that $\text{INFOGAP}[\mu, \nu]$ can be estimated to within $\varepsilon$ error using $\text{poly}(1/\varepsilon)$ observable prediction-outcome samples.
- Negative Result (Proposition 4.9): Any informativeness measure that is sound and complete must necessarily be discontinuous.
Limitations and Future Work
- Binary Outcomes Only: The theoretical results and characterizations in this workshop version are established for binary prediction tasks; extending the relaxed transport formulation to multi-class or continuous prediction spaces remains an open direction.
- Discontinuity in Practice: Because any consistent metric is inherently discontinuous, small sampling noise or slight perturbations in prediction distributions could cause sudden jumps in the estimated informativeness gap.
- Empirical Benchmarking: The work does not include real-world empirical evaluations comparing popular foundation models or modern calibration techniques (e.g., temperature scaling).
Summary updated