Is This Predictor More Informative than Another? A Decision-Theoretical Comparison

Yiding Feng, Weihua Tang, Tang, Wei

TL;DR

This paper introduces the Informativeness Gap (InfoGap), a new decision-theoretic metric to compare two predictive models—even when they are miscalibrated. By quantifying the maximum payoff advantage one model offers over another across all possible decision tasks, the authors provide a rigorous way to determine which "black-box" predictor is truly more useful for downstream users.

What Problem Does This Paper Address?

In machine learning, we often evaluate probabilistic forecasts using metrics like Expected Calibration Error (ECE) (how "honest" the probabilities are) or Brier Score (a mix of accuracy and calibration). However, these metrics don't always tell a model provider which model will be most useful for a user making real-world decisions.

A classic problem is that a perfectly calibrated model can be useless—imagine a weather forecaster who always predicts a 50% chance of rain because that is the long-run average. They are perfectly calibrated but provide zero information to help you decide whether to carry an umbrella. Conversely, a slightly miscalibrated model might be much more "informative" by correctly identifying high-risk days.

The central challenge is: How do you compare two models when neither is perfectly calibrated and you don't know the specific goals (payoff functions) of the people using them?

Methods

The authors propose a framework based on the Informativeness Gap (InfoGap).

  • The Core Metric: InfoGap measures the maximum difference in expected utility (payoff) between two predictors across all "normalized" decision tasks.
  • REMD (Relaxed Earth Mover's Distance): For perfectly calibrated models, they show that InfoGap is equivalent to a novel measure they call REMD. While the standard Earth Mover's Distance (Wasserstein-1) measures the cost of turning one distribution into another, REMD is a "relaxed" version that specifically captures decision-relevant differences.
  • CA-SCDF (Calibration-Adjusted Super-Cumulative Distribution Function): To handle miscalibrated models, they introduce a "calibration-adjusted" version of the SCDF. This allows them to visualize and calculate the InfoGap by looking at the maximum vertical separation between two curves.
  • InfoGap-Regularized Post-processing: They propose a new way to "fix" model calibration. Instead of just minimizing log-likelihood (which can sometimes make a model less useful for decisions), they add an InfoGap penalty to the optimization objective.

Datasets

The authors evaluated their theory using four popular Large Language Models (LLMs) acting as forecasters: GPT-4o mini, DeepSeek-V3.2, Gemini 2.0 Flash-Lite, and Qwen-Plus. They tested these models on two real-world domains:

  1. Weather: Predicting rain events using a public Kaggle meteorological dataset.
  2. Bitcoin: Predicting price increases (absolute and relative) using historical data from CoinMarketCap.

Key Insights and Ideas

  • The Informativeness Trilemma: The authors prove a fundamental "impossibility" result. No informativeness measure can simultaneously be complete (zero error for perfect models), sound (high error for bad models), and continuous (not sensitive to tiny perturbations). You have to pick two.
  • Generalizing the Classics: InfoGap is a "super-metric." It recovers Blackwell Informativeness (the gold standard in economics) when models are calibrated, and it subsumes recent metrics like U-Calibration and Calibration Decision Loss (CDL).
  • Calibration is a Double-Edged Sword: A major "aha" moment in the paper is that standard calibration post-processing (like Temperature Scaling) can actually decrease a model's usefulness. By trying to make the probabilities "honest," these methods can sometimes wash away the useful signal that a decision-maker relies on.
  • Decision-Relevant Quadrants: Using reliability diagrams, the authors show that models are most distinguishable at the "decision-flipping" thresholds—where a small change in probability changes the optimal action for the user.

Key Results

  • Resolving Metric Conflicts: In the weather task, Gemini had a better Brier score than Qwen, but Qwen had better ECE. Traditional metrics were at a stalemate. InfoGap broke the tie, showing Gemini had a significant decision-relevant advantage (0.215 vs 0.082).
  • Sample Efficiency: The authors proved that InfoGap can be estimated efficiently using only a modest number of samples ($O(1/\epsilon^2)$), making it practical for evaluating black-box APIs where you only see the final prediction and the outcome.
  • Better Recalibration: Their proposed InfoGap-regularized post-processing systematically reduced the "harm" caused by standard recalibration, particularly for Temperature Scaling, ensuring that the model stays useful while becoming more calibrated.

Limitations and Future Work

  • Binary Outcomes: The current framework is primarily designed for binary classification (Yes/No events). Extending this to multi-class or continuous outcomes is a significant next step.
  • The Continuity Trade-off: Because of the "Trilemma," the InfoGap is not continuous. This means it might be sensitive to very small changes in a model's output distribution, which could be a challenge for certain optimization loops.
  • Mechanism Design Links: The authors noted a potential deep connection between their "flow conservation" constraints and the field of mechanism design, suggesting that future work could bridge these two theoretical areas.

Summary updated

Abstract

In many real-world applications, a model provider provides probabilistic forecasts to downstream decision-makers who use them to make decisions under diverse payoff objectives. The provider may have access to multiple predictive models, each potentially miscalibrated, and must choose which model to deploy in order to maximize the usefulness of predictions for downstream decisions. A central challenge arises: how can the provider meaningfully compare two predictors when neither is guaranteed to be well-calibrated, and when the relevant decision tasks may differ across users and contexts? To answer this, our first contribution introduces the notion of the informativeness gap between any two predictors, defined as the maximum normalized payoff advantage one predictor offers over the other across all decision-making tasks. Our framework strictly generalizes several existing notions: it subsumes U-Calibration and Calibration Decision Loss, which compare a miscalibrated predictor to its calibrated counterpart, and it recovers Blackwell informativeness as a special case when both predictors are perfectly calibrated. Our second contribution is a dual characterization of the informativeness gap, which gives rise to a natural informativeness measure that can be viewed as a relaxed variant of the earth mover's distance between two prediction distributions. We show that this measure satisfies natural desiderata: it is complete and sound, and it can be estimated sample-efficiently in the prediction-only access setting. We complement our theory with experiments on LLM-based forecasters in real-world prediction tasks, showing that the informativeness gap offers a more decision-relevant alternative to traditional metrics, and provides a principled lens for evaluating how ad hoc calibration post-processing affects downstream decision usefulness.
BibTeX
@inproceedings{feng2025is,
  author = {Yiding Feng and Weihua Tang and Tang, Wei},
  title = {{Is This Predictor More Informative than Another? A Decision-Theoretical Comparison}},
  booktitle = {MLxOR @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=ohExiFlNme},
}
Figure