Abstract
Decision makers often need to rely on imperfect probabilistic forecasts. While average performance metrics are typically available, it is difficult to assess the quality of individual forecasts and the corresponding utilities. To convey confidence about individual predictions to decision-makers, we propose a compensation mechanism ensuring that the forecasted utility matches the actually accrued utility. While a naive scheme to compensate decision-makers for prediction errors can be exploited and might not be sustainable in the long run, we propose a mechanism based on fair bets and online learning that provably cannot be exploited. We demonstrate an application showing how passengers could confidently optimize individual travel plans based on flight delay probabilities estimated by an airline.
Introduction: The Calibration Gap
In high-stakes domains like personalized medicine or weather forecasting, we rely heavily on probabilistic predictions. A model might claim a 70% chance of success for a treatment, which is useful in aggregate—meaning it's right 70% of the time across a large population. However, for an individual patient, this "average calibration" provides little comfort. If the treatment fails for them, they bear the full cost of the error.
Achieving individual calibration—where a forecast is accurate for every single instance—is mathematically impossible without infinite data or providing uselessly broad confidence intervals. This paper proposes a paradigm shift: instead of trying to make every prediction "right," can we design a system where the decision-maker is protected when the prediction is "wrong"?
A Mechanism for Utility Preservation
The authors address this challenge through Mechanism Design, specifically by treating probabilities as a "willingness to bet."
The Betting Protocol
The proposed system introduces a Betting Protocol. The forecaster provides a probability ($\mu_t$) along with a confidence interval ($c_t$). The decision-maker (the agent) then selects an action and a "stake" ($b_t$). If the actual outcome deviates from the forecast, the forecaster provides a side payment to the agent. This acts as a form of insurance, ensuring that the agent's true expected utility aligns with the forecasted utility, regardless of the forecast's objective accuracy.
Sustainable Forecaster Strategies
To prevent the forecaster from being exploited by adversarial agents, the system employs Swap Regret Minimization. By analyzing past betting outcomes, the forecaster uses an online learning algorithm to adjust future probabilities and confidence intervals. This ensures "asymptotic exactness"—the forecaster’s total betting loss approaches zero over time. The confidence interval $c_t$ serves as a crucial buffer; if the forecaster has been losing money, $c_t$ increases, making the "insurance" more expensive and protecting the forecaster's budget.
Empirical Validation
The mechanism was tested across several diverse datasets:
- U.S. Flight Delay Data (DoT 2017): Real-world records used to simulate an airline offering "delay insurance" to passengers.
- MNIST: Benchmarking convergence on binary classification tasks derived from handwritten digits.
- UCI Adult Dataset: Testing robustness on standard tabular data.
Key Results
The simulations, particularly in the flight delay scenario, revealed significant benefits for both parties.
As shown in the results, the betting mechanism creates a "win-win" scenario. For airlines, the ability to offer "insured" forecasts increased revenue because risk-averse passengers were more willing to book flights. For passengers, the total utility increased as they could make decisions with the confidence that their realized utility would match the airline's predictions.
Crucially, the proposed adaptive algorithm consistently drove the forecaster's average betting loss toward zero within a few thousand iterations. Without this adaptive learning (as seen in the "Without Algorithm 2" plots), the system remained volatile or led to financial losses for the forecaster. Furthermore, the confidence intervals remained small and practical, ensuring the forecasts stayed "sharp" and the insurance remained affordable.
Conclusion and Future Directions
This research demonstrates that utility preservation is a viable and perhaps more practical alternative to the impossible goal of individual calibration. By framing forecasting as a mechanism design problem, we can build systems where users can trust and act upon probabilistic information without bearing the full risk of predictive error. Future work may explore extending these guarantees beyond binary outcomes and investigating the ethical implications of "buying" trust through compensation.
Summary updated