TL;DR
This paper reveals a critical flaw in "strictly proper" scoring rules, commonly used to incentivize honest probabilistic forecasts. When players can collude, these mechanisms can actually encourage them to lie about their beliefs to exploit arbitrage opportunities, leading to guaranteed profits for intermediaries and distorted information for decision-makers.
What Problem Does This Paper Address?
Decision-makers often rely on experts for probabilistic forecasts to navigate uncertainty. Strictly proper scoring rules (like Brier or logarithmic scores) are designed to incentivize these experts to report their true beliefs, ensuring the accuracy and reliability of the forecasts. Prediction markets, a popular application of these rules, aggregate individual forecasts to gauge collective belief. However, this paper investigates what happens when these independent forecasters can coordinate their actions – forming a "coalition." The core problem is that these seemingly robust mechanisms, designed for individual honesty, can be gamed by cooperating players, undermining their intended purpose.
Methods
The authors analyze traditional and competitive strictly proper scoring rules. They introduce the concept of "arbitrage" for a coalition of players, meaning there exists a coordinated reporting strategy that guarantees a higher payout for the coalition members than if they reported their true beliefs, regardless of the actual outcome. They mathematically demonstrate how this arbitrage arises for various scoring rules (quadratic, logarithmic, spherical) and for competitive mechanisms (like those proposed by Hanson, Kilgour & Gerchak, and Lambert). The proofs often involve showing that a specific coordinated report (an "identical report") yields a strictly positive surplus for the coalition compared to individual truthful reporting. They also explore the optimal size of such a coalition to maximize this surplus.
Datasets
Not discussed in the paper.
Key Insights and Ideas
- Cooperation Undermines Truthfulness: The most striking insight is that mechanisms designed for individual honesty can incentivize deception when players cooperate. This is counter-intuitive and highlights a fundamental vulnerability in these systems.
- Arbitrage as the Exploitation Mechanism: The paper formalizes how cooperating players can exploit scoring rules through arbitrage. By coordinating their reports, they can guarantee a better outcome for themselves than by acting independently.
- Intermediaries Profit from Collusion: The findings suggest that intermediaries (like web portals) could exploit this arbitrage by facilitating coalitions, guaranteeing themselves a profit without needing any forecasting expertise.
- Information Distortion: When players collude, they may present a false consensus, masking the true diversity of opinions among experts. This can mislead decision-makers who rely on this information.
- Vulnerability Across Scoring Rule Types: The arbitrage problem isn't limited to one type of scoring rule; it affects widely used rules like quadratic, logarithmic, and spherical scores, as well as competitive mechanisms.
Key Results
The paper proves that for various strictly proper scoring rules (quadratic, generalized logarithmic, spherical) and competitive mechanisms, coalitions of players can exploit arbitrage opportunities. This means they can coordinate their reports to achieve a higher payout than by reporting truthfully, especially when their beliefs differ. The paper provides specific mathematical conditions and examples for how this arbitrage occurs, demonstrating that the surplus from cooperation is strictly positive when coalition members disagree. For instance, in the case of quadratic scoring rules, the surplus is directly related to the variation in beliefs among coalition members.
Limitations and Future Work
- Focus on Arbitrage, Not Maximal Payment: The authors primarily demonstrate the possibility of guaranteed profit (arbitrage) rather than optimizing for maximal expected payment. They suggest that an intermediary maximizing their own payment might use their own forecast for the coalition.
- Assumptions on Coalition Behavior: The paper assumes players within a coalition act strategically to maximize their collective gain. It acknowledges that the internal dynamics of how coalition members share the surplus and communicate are complex and were studied in separate work (Chun, 2011).
- Difficulty in Detection: The paper notes that detecting and preventing cooperation among players in prediction markets is difficult, and intermediaries could exploit this.
- Open Question: Truly Robust Mechanisms: A key open question is whether any strictly proper mechanism exists that is immune to cooperation-induced arbitrage. The authors express doubt about this possibility.
- Future Work: The authors suggest further research into how coalition members behave internally, how to design mechanisms resistant to cooperation, and the implications for decision-makers who rely on these forecasts. They also touch upon the idea of using a quadratic scoring rule, which encourages reporting the weighted average, as a potential way to mitigate some of the negative effects of cooperation.
Summary updated