In a standard forecasting competition, the person with the highest score wins. This "Simple Max" mechanism seems intuitive: if you want to find the best forecaster, you should reward the one who is most accurate. However, research by Jens Witkowski, David Pennock, and their colleagues has shown that this format is fundamentally broken.
When a prize is awarded only to the top rank, participants are no longer incentivized to report their true beliefs. Instead, they are incentivized to gamble.
The Problem: Extremization and Hedging
The core issue is a misalignment between maximizing an expected score and maximizing the probability of winning.
In Incentive-Compatible Forecasting Competitions, the authors demonstrate that if a forecaster is behind in the rankings, their best strategy is often to "extremize." By reporting a probability of 0% or 100% for an unlikely event, they increase their variance. If that unlikely event occurs, they leapfrog the leaders; if it doesn't, they were going to lose anyway. This strategic noise pollutes the aggregate data, making the competition less useful for actual prediction.
Conversely, as shown in Hedging and Approximate Truthfulness in Traditional Forecasting Competitions by Anish Thilagar and Rafael Frongillo, leaders face the opposite incentive: they "hedge." To protect a lead, a top-ranked forecaster might report more moderate, consensus-driven beliefs to minimize the risk of a single bad outcome wiping out their advantage.
The Solution: The ELF Mechanism
To fix this, Rupert Freeman, David Pennock, and others proposed the Event-Lotteries Forecaster Selection Mechanism (ELF).
The intuition behind ELF is to replace the deterministic "winner-take-all" rule with a randomized one. Instead of the highest score winning automatically, the winner is chosen via a lottery where each participant’s chance of winning is proportional to their score (calculated using a proper scoring rule).
Because your probability of winning is now a linear function of your score, the strategy that maximizes your expected score—reporting your true belief—is exactly the same strategy that maximizes your chance of winning the prize. This makes the competition incentive-compatible.
Scaling to Real-World Complexity
While ELF works for independent events, real-world forecasting often involves correlated events (e.g., multiple states in an election) or performative predictions (where the forecast itself changes the outcome).
- Correlations: In Forecasting Competitions with Correlated Events, Rafael Frongillo and Bo Waggoner introduced the $(b, c)$-block correlation framework. They proved that mechanisms like Follow-the-Regularized-Leader (FTRL) remain truthful even when events influence one another, provided the number of events is sufficiently large to overcome the noise of the correlations.
- Performative Feedback: A more subtle problem arises when a prediction influences the world. For example, if an AI predicts a stock market crash, investors might sell, causing the crash. Caspar Oesterheld and Johannes Treutlein argue in Incentivizing honest performative predictions with proper scoring rules that standard scoring rules fail here. Agents are incentivized to report forecasts that manipulate the world into a state that is easier to predict, rather than reporting the truth. Philip Boeken and colleagues suggest that "observationally proper" rules must be developed to account for this causal link between the forecast and the outcome.
Beyond Accuracy: Reputation and Noise
Finally, incentives aren't always about prize money. Andrew Gelman and Jessica Hullman point out that in election forecasting, the incentive is often reputational. Forecasters may intentionally inflate uncertainty intervals to avoid the "embarrassment" of a result falling outside their predicted range.
To combat the inherent noise of real-world outcomes, new mechanisms like WOMAC, introduced by Connacher Murphy and Yiling Chen in WOMAC: A Mechanism For Prediction Competitions, seek to further refine how we identify top talent by accounting for the variance that standard leaderboards ignore.
By moving from simple rankings to mathematically grounded mechanisms like ELF and FTRL, researchers are building systems where the best way to win is simply to tell the truth.