Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment

Ivan Stelmakh, Nihar B. Shah, Aarti Singh

Introduction

Peer assessment is a cornerstone of modern evaluation systems, used extensively in academic grading, hiring processes, and conference reviews. However, its integrity is often threatened by strategic manipulation. Reviewers, who are also being evaluated in the same pool, may be tempted to downrank their competitors to improve their own standing. While detection methods exist for rating-based systems (where reviewers assign absolute scores), ranking-based systems—where reviewers order submissions from best to worst—present a unique challenge. This paper introduces a novel, non-parametric statistical test designed specifically to detect such strategic behavior in ranking systems.

Methodology: A Non-Parametric Approach

The authors frame the detection of strategic behavior as a non-parametric statistical independence test. The core idea is to determine if a reviewer's submitted ranking is independent of their own submission's standing.

The Null Hypothesis and Test Statistic

The null hypothesis assumes the reviewer is grading honestly. The test statistic measures the "gain"—the impact of a reviewer's submitted ranking on the final standing of their own work. This is compared against the expected standing if the reviewer had submitted a uniformly random ranking.

Permutation Testing

To determine if a test statistic is suspiciously high, the algorithm generates a null distribution by permuting the "authorship matrix" (who authored what) while keeping the assignment graph (who reviewed what) fixed. If the actual outcome benefits the reviewer significantly more than these permuted alternate realities, the test rejects the null hypothesis.

The Role of Supervision

The test can operate in two modes:

  • Unsupervised: The test relies purely on the peer-submitted rankings.
  • Supervised: The test incorporates an optional set of "impartial rankings" (e.g., from a teaching assistant or expert). This supervision isolates the manipulator's impact and dramatically boosts detection power.

Experimental Evaluation

Because real-world data lacks ground-truth labels for manipulation, the authors conducted a gamified experiment with 55 graduate students. Participants were explicitly incentivized to manipulate rankings to boost their own hypothetical submissions. In the first four rounds, participants played against truthful bots, while the fifth round involved players competing against each other.

Results and Key Insights

The proposed test demonstrates strict control over the false alarm rate while providing significant detection power, especially when supervision is available.

Detection power across five rounds of the peer-assessment game.

As shown in the table above, the test reliably detected manipulation in the human experiments when provided with impartial ground-truth rankings, achieving detection rates as high as 100% in some rounds. Without supervision, the power is lower because the baseline is muddied by other manipulators, but it still provides a statistically significant signal.

Detection power as a function of the fraction of truthful reviewers, with and without supervision.

The power curves illustrate several key insights:

  • Supervision is a Force Multiplier: Having access to even a small amount of ground-truth data establishes a clean baseline, making deviations obvious.
  • Better Cheaters are Easier to Catch: In semi-synthetic trials, the authors found that as participants learned to manipulate more effectively, the test's detection power actually increased because the resulting gain became more statistically distinct.
  • Dynamic Cheating: Unlike rating systems, ranking systems force strategic reviewers to adapt. A strategy that works when a paper is ranked 2nd might fail if it is ranked 4th. The proposed test cleverly avoids modeling these complex strategies by focusing purely on the outcome.

Conclusion and Future Work

This research provides a robust tool for organizers to detect manipulation in ranking-based peer assessment. By focusing on the outcome rather than the intent, the test remains flexible and effective. A primary limitation is that it currently detects successful manipulations—those that actually move the needle. Future work will focus on increasing sensitivity to detect any manipulation attempt, regardless of its success in gaming the system.

Summary updated

BibTeX
@inproceedings{stelmakh2021catch,
  author = {Ivan Stelmakh and Nihar B. Shah and Aarti Singh},
  title = {{Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i6.16611},
}
Figure