Scoring Rules for Performative Binary Prediction

Alan ChanarXiv 2022 · OpenReview

TL;DR

This paper explores a critical issue in AI: what happens when predictions change the reality they're predicting (performative prediction)? They show that standard "proper scoring rules," designed to incentivize honest predictions, can actually encourage experts to manipulate the world to maximize their rewards. The good news? They propose a simple class of scoring rules that does incentivize experts to report truthfully, even in performative settings.

What Problem Does This Paper Address?

In many real-world applications, predictions aren't just passive observations; they actively influence the outcome. Think of predicting inflation: if everyone believes inflation is coming, they'll buy goods now, which causes inflation. This is "performative prediction."

Traditionally, "proper scoring rules" are used to reward experts for accurate predictions. The idea is that if an expert is rewarded based on how well their prediction matches the outcome, they'll be incentivized to be accurate. However, this assumes predictions don't affect the outcome. When predictions are performative, a clever expert might realize they can manipulate the outcome to get a better score, even if they're using a "proper" scoring rule. This paper tackles this fundamental conflict: how do we design scoring rules that incentivize accurate predictions when those predictions themselves can alter reality?

Methods

The authors build a formal model of performative binary prediction. This model incorporates:

  • Performativity: They introduce two ways predictions can influence the true probability of an event:
  • Drift: The prediction directly shifts the probability towards itself (e.g., a prediction of high inflation leads to actions that cause high inflation).
  • Reversion: Extreme predictions (close to 0 or 1) push the probability towards a more neutral state (e.g., predicting a landslide victory might reduce voter turnout, making the election closer).
  • Audits: They acknowledge that experts aren't entirely unaccountable. They model an "audit" mechanism where experts face a cost if they are found to have misreported, with the probability of audit and cost potentially depending on how far the prediction is from the true outcome.
  • Expected Reward: The expert's goal is to maximize their expected reward, which is a function of their prediction, the true outcome, the scoring rule, the performativity model, and the audit cost.

They then analyze whether standard "strictly proper scoring rules" (like quadratic, spherical, and logarithmic) remain "incentive-compatible" (meaning the expert's best strategy is to predict the true probability) under these performative models. They also propose and analyze a new class of scoring rules.

Datasets

Not discussed in the paper. The paper focuses on theoretical modeling and numerical simulations rather than empirical evaluation on real-world datasets.

Key Insights and Ideas

  • Proper Scoring Rules Can Backfire: The most striking finding is that standard, strictly proper scoring rules fail to prevent manipulation in performative settings. An expert can exploit the performativity to achieve a higher reward, even if it means deviating from the true probability.
  • Performativity Models Matter: The way predictions influence reality (drift vs. reversion) significantly impacts the expert's optimal strategy and how much they deviate from the true probability.
  • Audits Help, But Don't Solve Everything: While audits and associated costs can reduce the incentive to manipulate, they don't fully restore incentive compatibility for strictly proper scoring rules. The expert's optimal prediction still deviates from the true probability.
  • A Simple Solution Exists: The paper identifies a straightforward class of scoring rules – constant scoring rules – that are incentive-compatible in performative settings, provided the audit cost is structured appropriately. This is a crucial positive result.

Key Results

  • Theoretical Proofs: The paper mathematically proves that for both drift and reversion models, strictly proper scoring rules are not incentive-compatible when performativity is present (i.e., when the prediction influences the outcome). The expert's optimal prediction will deviate from the true probability.
  • Numerical Simulations: Figures in the paper illustrate these findings. They show how the expert's optimal prediction deviates from the true probability for different scoring rules (quadratic, spherical, log), performativity parameters (drift a, reversion q), and audit costs (c). Generally, higher audit costs bring the expert's prediction closer to the true probability, but a gap often remains. The degree of performativity (a in the drift model) also strongly influences the deviation.
  • Incentive-Compatible Rule: The authors demonstrate that a constant scoring rule (where the reward is the same regardless of the prediction) combined with an audit cost that penalizes deviations from the true probability can ensure incentive compatibility.

Limitations and Future Work

  • Imposing Audit Costs on Algorithmic Systems: A significant limitation is the practical challenge of implementing audit costs for algorithmic systems. The paper acknowledges that it's unclear how to impose such costs on AI agents.
  • Defining "Manipulation": The paper assumes any deviation from the true probability is manipulation. However, in truly performative systems, the expert might have no choice but to change the world. The authors suggest a more nuanced question: what manipulation is most beneficial for the user?
  • Expert's Knowledge of True Probability: The current model assumes experts know the true probability distribution. In reality, experts might need to learn this distribution. Future work could explore scoring rules that incentivize both learning and disincentivize manipulation.
  • Audit Cost Structure: The assumed audit cost is proportional to the difference between the prediction and the true probability. More complex audit cost structures might be necessary or more realistic in certain scenarios.

Summary updated

Abstract

We construct a model of expert prediction where predictions can influence the state of the world. Under this model, we show through theoretical and numerical results that proper scoring rules can incentivize experts to manipulate the world with their predictions. We also construct a simple class of scoring rules that avoids this problem.
BibTeX
@misc{chan2022scoring,
  author = {Alan Chan},
  title = {{Scoring Rules for Performative Binary Prediction}},
  year = {2022},
  eprint = {2207.02847},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.02847},
}
Figure