Abstract
Evaluating predictive models typically relies on calibration metrics like Expected Calibration Error (ECE) or scoring rules like the Brier score. However, these metrics often fail to reflect the actual utility of a model for downstream decision-makers, particularly when models are miscalibrated or when the specific decision tasks are unknown. We propose the Informativeness Gap (InfoGap), a decision-theoretic metric that quantifies the maximum payoff advantage one predictor offers over another across all possible normalized decision tasks. Our framework handles miscalibrated models through a novel characterization called REMD-MisC and introduces the Calibration-Adjusted Super-Cumulative Distribution Function (CA-SCDF) for visual analysis. Experiments with state-of-the-art Large Language Models (LLMs) on weather and financial data demonstrate that InfoGap provides a more robust and decision-relevant ranking than traditional metrics.
Introduction
In machine learning, "calibration"—the alignment of predicted probabilities with observed frequencies—is often treated as a primary goal. Yet, calibration is frequently a poor proxy for utility. A perfectly calibrated model can be useless if it lacks "sharpness" (e.g., always predicting the base rate), while a miscalibrated model might provide significantly more actionable information.
Existing metrics suffer from two main issues:
- Self-Referentiality: Metrics like ECE measure distance from a model's own perfectly calibrated version, making it difficult to compare two distinct models.
- Decision Irrelevance: They do not directly translate to the payoff a user receives in a real-world task.
This paper addresses these gaps by asking: Which predictor should be deployed to maximize payoff for users with diverse, unknown goals?
Methodology: The Informativeness Gap
We define the Informativeness Gap (InfoGap) as the maximum payoff advantage one predictor provides over another across the space of all possible "normalized" decision tasks.
Dual Characterization
For perfectly calibrated models, we prove that InfoGap is equivalent to a Relaxed Earth Mover’s Distance (REMD) between prediction distributions. To account for real-world miscalibration, we extend this to REMD-MisC, which incorporates the true conditional frequencies of outcomes given predictions.
Visualizing Utility with CA-SCDF
We introduce the Calibration-Adjusted Super-Cumulative Distribution Function (CA-SCDF). This tool allows researchers to visualize the InfoGap as the maximum vertical distance between two curves. If the curves for two models intersect, it indicates that one model is superior for low-threshold decisions while the other excels at high-threshold tasks.
Experimental Setup
We evaluated the InfoGap framework using four LLMs—ChatGPT (GPT-4o mini), DeepSeek-V3.2, Gemini 2.0 Flash-Lite, and Qwen-Plus—across two forecasting domains:
- Weather Rain Dataset: Predicting single-day and two-day rain events using Kaggle meteorological data.
- Bitcoin Price Dataset: Predicting price movements across various thresholds using historical OHLCV data.
Key Insights
The Informativeness Trump Card
Our analysis shows that informativeness often outweighs perfect calibration. A "sharp" but miscalibrated predictor can be strictly better for every possible decision-maker than a "flat" but perfectly calibrated one.
The Continuity Trilemma
We establish a theoretical "impossibility result": no informativeness measure can be simultaneously complete (correctly ranking all models), sound (never misidentifying a better model), and continuous (stable under small perturbations). InfoGap prioritizes completeness and soundness, accepting inherent discontinuities.
Calibration Can Reduce Utility
Standard post-processing techniques like Temperature Scaling, while improving ECE, can inadvertently "wash away" useful information. We propose an InfoGap-regularized objective for recalibration that balances likelihood with decision utility, significantly reducing this "utility harm."
Results
- Resolving Ambiguity: In the Bitcoin price prediction task, ChatGPT and Gemini showed nearly identical ECE scores. InfoGap, however, revealed a significant utility advantage for Gemini, providing a clear separation where traditional metrics failed.
- Sample Efficiency: InfoGap can be estimated efficiently from prediction-label pairs alone, making it suitable for "black-box" model evaluation.
- Improved Recalibration: Our regularized approach consistently outperformed standard methods in preserving model utility during the calibration process.
Conclusion and Future Work
InfoGap offers a rigorous, decision-theoretic alternative to traditional calibration metrics. While currently limited to binary outcomes and subject to the continuity trilemma, it provides a more faithful measure of a model's real-world value. Future work will focus on extending the framework to multi-class settings and exploring deeper connections with mechanism design.
Summary updated