Abstract
Traditional peer prediction mechanisms, designed for low-dimensional data like star ratings, struggle with the complexity of textual feedback. This paper introduces novel mechanisms that leverage Large Language Models (LLMs) to incentivize high-effort, informative text reports. By treating LLMs as "oracles" that predict one reviewer's content based on another's, we develop scoring rules that reward genuine insight and penalize generic or AI-generated "fluff." Evaluations on Yelp and ICLR datasets demonstrate that these mechanisms successfully distinguish human expertise from automated summaries.
Introduction: The Textual Crisis
In systems like academic peer review or product ratings, high-quality subjective feedback is essential but difficult to verify without ground truth. Peer prediction mechanisms traditionally solve this by rewarding agents whose reports correlate with their peers. However, the rise of LLMs has created a "textual crisis": it is now trivial to generate professional-sounding but shallow reviews. Simple metrics like word count can no longer distinguish an expert human insight from a generic GPT-4 summary. This research asks: Can we automate incentives that reward deep, informative text over generic content?
The Peer Prediction Framework
The core idea is to reward reviewers based on how much their report helps an LLM predict the specific content of a peer's report.
As shown in Figure 1, an item $z$ generates signals for multiple reviewers. These reviewers provide reports $\tilde{X}$, which are processed by a mechanism $M$ to produce performance scores $s$. This framework allows for scoring without needing an external ground truth.
Methods: GPPM and GSPPM
The authors propose two primary mechanisms that leverage the probabilistic nature of LLMs:
- Generative Peer Prediction Mechanism (GPPM): Rewards Reviewer A based on how much their review reduces the LLM's "surprise" (log-probability) when predicting Reviewer B’s report.
- Generative Synopsis Peer Prediction Mechanism (GSPPM): An advanced version that rewards information provided beyond a simple synopsis (like an abstract). This prevents reviewers from gaining high scores just for summarizing the work.
To implement these, they use token-based measurements of log-probabilities and judgment-based approaches where the LLM distills reviews into specific claims to be cross-referenced.
The Challenge of Effort
A key insight is that the high dimensionality of text, while complex, actually makes it easier for LLMs to identify high-effort signals.
Figure 2 illustrates how different effort levels produce varying signals. A "high effort" signal contains specific, non-obvious details that are much more useful for predicting a peer's independent observation than a "low effort" generic statement. The mechanisms are designed to reward the transition from generic "fluff" to specific, informative claims.
Key Results
The mechanisms were evaluated on Yelp reviews and ICLR 2020 OpenReview data, alongside synthetic AI reviews generated by GPT-3.5 and GPT-4.
- Differentiating Quality: On the ICLR dataset, the mechanisms successfully ranked quality as Human > GPT-4 > GPT-3.5. Human reviews received the highest expected scores, reflecting their deeper insight.
- Penalizing AI "Fluff": GSPPM was particularly effective at penalizing AI-generated reviews. Because AI often sticks close to the provided abstract, GSPPM—which conditions on the abstract—identified the lack of novel information.
- Sensitivity: These text-based mechanisms were far more sensitive to quality than traditional baselines that only looked at numerical 1-5 star ratings.
Conclusion and Future Work
By using LLMs as proxies for collective human knowledge, these mechanisms provide a path toward automated, scalable incentive systems for high-quality text. While promising, future work must address the computational cost of multiple LLM calls and the potential for malicious actors to attempt to "game" the scoring LLM with specific prompts.
Summary updated