TL;DR
This paper introduces the Informativeness Gap (InfoGap), a new decision-theoretic metric to compare two predictive models—even when they are miscalibrated. By quantifying the maximum payoff advantage one model offers over another across all possible decision tasks, the authors provide a rigorous way to determine which "black-box" predictor is truly more useful for downstream users.
What Problem Does This Paper Address?
In machine learning, we often evaluate probabilistic forecasts using metrics like Expected Calibration Error (ECE) (how "honest" the probabilities are) or Brier Score (a mix of accuracy and calibration). However, these metrics don't always tell a model provider which model will be most useful for a user making real-world decisions.
A classic problem is that a perfectly calibrated model can be useless—imagine a weather forecaster who always predicts a 50% chance of rain because that is the long-run average. They are perfectly calibrated but provide zero information to help you decide whether to carry an umbrella. Conversely, a slightly miscalibrated model might be much more "informative" by correctly identifying high-risk days.
The central challenge is: How do you compare two models when neither is perfectly calibrated and you don't know the specific goals (payoff functions) of the people using them?
Methods
The authors propose a framework based on the Informativeness Gap (InfoGap).
- The Core Metric: InfoGap measures the maximum difference in expected utility (payoff) between two predictors across all "normalized" decision tasks.
- REMD (Relaxed Earth Mover's Distance): For perfectly calibrated models, they show that InfoGap is equivalent to a novel measure they call REMD. While the standard Earth Mover's Distance (Wasserstein-1) measures the cost of turning one distribution into another, REMD is a "relaxed" version that specifically captures decision-relevant differences.
- CA-SCDF (Calibration-Adjusted Super-Cumulative Distribution Function): To handle miscalibrated models, they introduce a "calibration-adjusted" version of the SCDF. This allows them to visualize and calculate the InfoGap by looking at the maximum vertical separation between two curves.
- InfoGap-Regularized Post-processing: They propose a new way to "fix" model calibration. Instead of just minimizing log-likelihood (which can sometimes make a model less useful for decisions), they add an InfoGap penalty to the optimization objective.
Datasets
The authors evaluated their theory using four popular Large Language Models (LLMs) acting as forecasters: GPT-4o mini, DeepSeek-V3.2, Gemini 2.0 Flash-Lite, and Qwen-Plus. They tested these models on two real-world domains:
- Weather: Predicting rain events using a public Kaggle meteorological dataset.
- Bitcoin: Predicting price increases (absolute and relative) using historical data from CoinMarketCap.
Key Insights and Ideas
- The Informativeness Trilemma: The authors prove a fundamental "impossibility" result. No informativeness measure can simultaneously be complete (zero error for perfect models), sound (high error for bad models), and continuous (not sensitive to tiny perturbations). You have to pick two.
- Generalizing the Classics: InfoGap is a "super-metric." It recovers Blackwell Informativeness (the gold standard in economics) when models are calibrated, and it subsumes recent metrics like U-Calibration and Calibration Decision Loss (CDL).
- Calibration is a Double-Edged Sword: A major "aha" moment in the paper is that standard calibration post-processing (like Temperature Scaling) can actually decrease a model's usefulness. By trying to make the probabilities "honest," these methods can sometimes wash away the useful signal that a decision-maker relies on.
- Decision-Relevant Quadrants: Using reliability diagrams, the authors show that models are most distinguishable at the "decision-flipping" thresholds—where a small change in probability changes the optimal action for the user.
Key Results
- Resolving Metric Conflicts: In the weather task, Gemini had a better Brier score than Qwen, but Qwen had better ECE. Traditional metrics were at a stalemate. InfoGap broke the tie, showing Gemini had a significant decision-relevant advantage (0.215 vs 0.082).
- Sample Efficiency: The authors proved that InfoGap can be estimated efficiently using only a modest number of samples ($O(1/\epsilon^2)$), making it practical for evaluating black-box APIs where you only see the final prediction and the outcome.
- Better Recalibration: Their proposed InfoGap-regularized post-processing systematically reduced the "harm" caused by standard recalibration, particularly for Temperature Scaling, ensuring that the model stays useful while becoming more calibrated.
Limitations and Future Work
- Binary Outcomes: The current framework is primarily designed for binary classification (Yes/No events). Extending this to multi-class or continuous outcomes is a significant next step.
- The Continuity Trade-off: Because of the "Trilemma," the InfoGap is not continuous. This means it might be sensitive to very small changes in a model's output distribution, which could be a challenge for certain optimization loops.
- Mechanism Design Links: The authors noted a potential deep connection between their "flow conservation" constraints and the field of mechanism design, suggesting that future work could bridge these two theoretical areas.
Summary updated