Measurement Integrity in Peer Prediction: A Peer Assessment Case Study

Noah Burrell, Grant Schoenebeck

TL;DR

This paper introduces measurement integrity—the ability of a mechanism to fairly and accurately reward participants based on the actual quality of their reports—as a crucial new desideratum for peer prediction systems. Through both agent-based simulations and real-world peer-grading data, the authors reveal a stark trade-off between measurement integrity and strategic robustness, demonstrating that incorporating domain-specific parametric statistical models can successfully bridge this gap.

What Problem Does This Paper Address?

Peer prediction mechanisms are designed to elicit honest information from a population when there is no objective "ground truth" available for immediate verification (e.g., crowdsourcing, peer grading). Historically, the theoretical literature has focused almost exclusively on robustness against strategic reporting—ensuring that telling the truth forms a game-theoretic equilibrium.

However, in practical applications like "meta-grading" (grading the graders) in university courses or Massive Open Online Courses (MOOCs), ex post fairness is just as critical. Participants care deeply about whether their actual reward accurately reflects their effort and quality, not just if the system is fair "in expectation." This paper addresses the lack of metrics and frameworks for evaluating how well peer prediction mechanisms actually measure and reward individual report quality in data-scarce environments.

Methods

The authors formalize "measurement integrity" by analyzing the mathematical relationship between an agent's report quality and their final payment. They evaluate mechanisms using several statistical scales:

  • Coarse Ordinal: Binary and quinary classification using Area Under the Curve (AUC).
  • Fine Ordinal: Ranking accuracy using Kendall’s $\tau_B$.
  • Beyond Ordinal: Linear relationships using the Pearson correlation coefficient ($\rho$).

They test these properties using two complementary approaches:

  1. Agent-Based Modeling (ABM): A simulated classroom environment where students possess heterogeneous grading skills, varying effort levels, and distinct grading biases.
  2. Mechanism Augmentation: They compare standard non-parametric mechanisms—like Output Agreement (OA) and Peer Truth Serum (PTS)—against parametric mechanisms augmented with a statistical model (specifically, the PG1 model from Piech et al.). This parametric approach uses an Expectation-Maximization (EM) style algorithm to dynamically estimate and correct for latent grader biases and reliabilities.

Datasets

The authors validate their simulated findings using a real-world peer grading dataset:

  • Northwestern University Algorithms Course: Data spanning four semesters (Spring 2017, Fall 2017, Spring 2019, Fall 2019). It contains peer-submitted grades, instructor grades (serving as the ground truth proxy), and student comments, all coarsened to a standard $[0, 10]$ scale for evaluation.

Key Insights and Ideas

  • The Integrity-Robustness Frontier: There is an inherent trade-off between a mechanism's fairness and its security. Simple baselines like Mean Squared Error (MSE) provide excellent measurement integrity but are highly vulnerable to strategic manipulation (e.g., students coordinating on a single lazy score). Conversely, standard peer prediction mechanisms are strategically robust but fail to reward students fairly based on actual performance.
  • Parametric Shifting: Generic, out-of-the-box peer prediction mechanisms perform poorly because they require massive amounts of data to estimate report distributions. Supplementing these mechanisms with parametric statistical models allows them to individualize parameters even with sparse data, significantly shifting the Pareto frontier upward.
  • Theory vs. Empirical Reality: Traditional game-theoretic equilibrium concepts (such as $\epsilon$-strongly truthful) are noisy predictors of how a mechanism will actually perform when faced with simple, intuitive real-world deviations like hedging (grading closer to the prior mean to play it safe).

Key Results

  • The Best of Both Worlds: Parametric mechanisms, specifically $\Phi\text{-Div}_P: \text{KL}$ and $\Phi\text{-Div}_P: H^2$, strike the most effective balance on the frontier. As shown in the aggregated results, $\Phi\text{-Div}_P: \text{KL}$ maintains high measurement integrity while ensuring that the empirical rank gain from strategic cheating is essentially zero or negative.
Direct, two-dimensional comparisons of peer prediction mechanisms showing the trade-off between measurement integrity and strategic robustness across ABM and Real Data.
  • Baseline Performance: Standard non-parametric peer prediction mechanisms from literature largely fail to outperform simple baselines in measurement integrity, highlighting why they have seen slow adoption in real-world educational tools.

Limitations and Future Work

  • Model Dependency: The success of the parametric approach relies on having a reasonably accurate underlying model of the domain (e.g., grading behavior). In entirely novel domains where human behavior cannot be well-modeled, performance might degrade.
  • Collusion on Superficial Signals: The current framework does not account for settings where agents coordinate on "cheap," superficial signals (like grading based on the presence of typos rather than content quality).
  • Future Directions: The authors suggest extending empirical measurement integrity evaluations to other information elicitation contexts, integrating expert spot-checking, and exploring how machine learning agents adapt to these mechanisms over time.

Summary updated

Abstract

We propose measurement integrity, a property related to ex post reward fairness, as a novel desideratum for peer prediction mechanisms in many natural applications. Like robustness against strategic reporting, the property that has been the primary focus of the peer prediction literature, measurement integrity is an important consideration for understanding the practical performance of peer prediction mechanisms. We perform computational experiments, both with an agent-based model and with real data, to empirically evaluate peer prediction mechanisms according to both of these important properties. Our evaluations simulate the application of peer prediction mechanisms to peer assessment -- a setting in which ex post fairness concerns are particularly salient. We find that peer prediction mechanisms, as proposed in the literature, largely fail to demonstrate significant measurement integrity in our experiments. We also find that theoretical properties concerning robustness against strategic reporting are somewhat noisy predictors of empirical performance. Further, there is an apparent trade-off between our two dimensions of analysis. The best-performing mechanisms in terms of measurement integrity are highly susceptible to strategic reporting. Ultimately, however, we show that supplementing mechanisms with realistic parametric statistical models can, in some cases, improve performance along both dimensions of our analysis and result in mechanisms that strike the best balance between them.
BibTeX
@inproceedings{burrell2021measurement,
  author = {Noah Burrell and Grant Schoenebeck},
  title = {{Measurement Integrity in Peer Prediction: A Peer Assessment Case Study}},
  booktitle = {EC},
  year = {2021},
  doi = {10.1145/3580507.3597744},
}
Figure