TL;DR
This paper introduces swissnoise, an experimental platform for conducting online opinion polls using game-theoretic incentives. It compares traditional prediction markets with a novel peer prediction scheme, demonstrating that peer prediction can achieve comparable accuracy while being applicable to a broader range of questions.
What Problem Does This Paper Address?
Gathering accurate information from a large group of individuals, especially for subjective or future events, is challenging. Traditional polls often suffer from biased responses due to self-selection and the lack of strong incentives for participants to provide honest, well-informed answers. While prediction markets have shown promise in incentivizing accurate predictions for verifiable outcomes, they face legal hurdles (often treated as gambling) and are not suitable for questions without a clear, verifiable "ground truth," such as product quality or subjective opinions. This paper addresses the need for a practical, scalable, and legally sound mechanism to elicit truthful information for a wider array of questions.
Methods
The core contribution is the swissnoise platform, which implements and compares two incentive mechanisms:
- Prediction Markets: The paper utilizes a logarithmic market scoring rule, where participants trade securities linked to outcomes. The price of these securities reflects the market's collective probability assessment. An automated market maker adjusts prices to simulate liquidity.
- Peer Prediction (Peer Truth Serum): This is the novel contribution. Inspired by existing peer prediction schemes, swissnoise adapts them for public opinion polls. Instead of relying on a verifiable ground truth, it uses the reports of other participants as a proxy. The "peer truth serum" rewards participants when their report matches a randomly selected peer report from the same period. The reward is inversely proportional to the current estimated probability of that outcome, incentivizing honest reporting. This scheme is designed to be robust to variations in prior beliefs by using the current poll's aggregated probability as a common prior.
Datasets
The paper does not explicitly list or introduce new datasets for training. Instead, it uses real-world data collected from the swissnoise platform itself. The platform was launched on April 22nd, 2013, and by January 27th, 2014, it had over 200 active users and 132 open events. The evaluation also compares results against traditional opinion poll data from gfs.bern for three specific Swiss ballot items.
Key Insights and Ideas
- Peer Prediction is a Viable Alternative: The most significant insight is that peer prediction schemes, adapted for online polls, can achieve accuracy comparable to prediction markets, even for questions without a clear verifiable outcome.
- Broader Applicability: Peer prediction's ability to function without a verifiable ground truth makes it suitable for a much wider range of questions than prediction markets, opening up new possibilities for crowdsourced information.
- Incentive Design Matters: The paper highlights how initial reward scaling in peer prediction led to risk-seeking behavior and daily oscillations for binary outcomes. Adjusting the reward mechanism (e.g., using a running statistic over 5 days) and limiting ticket purchases helped mitigate these issues.
- User Behavior is Complex: Users exhibit strategic behaviors, like timing trades to maximize weekly profits in prediction markets. In peer prediction, users can adapt their opinions based on external information like traditional polls.
- Privacy is Key: The design of swissnoise emphasizes user privacy, preventing users from seeing each other's current or past opinions, which is crucial for encouraging honest reporting, especially in contexts like Switzerland.
Key Results
- Comparable Accuracy: Figure 4 shows that for events with a predicted probability of 0.5 or more, peer prediction achieved 72% accuracy, while prediction markets achieved 62%. This indicates similar performance, with peer prediction showing a slight edge.
- Profit Distribution: Figure 3(a) shows that the median user on the prediction market improved the market's forecast accuracy with a median profit of 1.47. Figure 3(b) illustrates user returns, with peaks indicating users who lost money, slightly improved accuracy, or significantly improved accuracy through risk-seeking behavior.
- Liquidity Impact: Figure 5 demonstrates that forecast accuracy in prediction markets is sensitive to liquidity parameters, with optimal values varying significantly across different events.
- User Activity: Figure 6 shows user activity, with prediction markets seeing more trades and peer prediction events seeing more votes, indicating engagement in both mechanisms.
Limitations and Future Work
- Representativeness: The authors acknowledge that swissnoise users might not be representative of the general Swiss population, although they are related to Switzerland and follow local media.
- Comparison with Traditional Polls: Direct comparison with traditional opinion polls is challenging because the questions asked differ fundamentally (e.g., "what do you think the outcome will be?" vs. "how will you vote?").
- Future Work: The authors plan to study user behavior with hypothetical questions under peer prediction incentives and explore adaptive liquidity parameters for prediction markets. They also aim to further investigate the practical implementation and design choices for peer prediction in online polls.
Summary updated