Solving reviewer assignment problem in software peer review: An approach based on preference matrix and asymmetric TSP model

Yanqing Wang, Yu Jiang, Xiaolei Wang, Siyu Zhang, Yaowen Liang

Abstract

Optimized reviewer assignment can effectively utilize limited intellectual resources and significantly assure review quality in various scenarios such as paper selection in conferences or journals, and proposal selection in funding agencies. However, little research on reviewer assignment for software peer review has been found. In this study, an optimization approach is proposed based on students' preference matrices and the model of the Asymmetric Traveling Salesman Problem (ATSP). Due to the critical role of the rule matrix in this approach, we conducted a questionnaire to obtain students' preference matrices and converted them to rule matrices. Using IBM ILOG CPLEX, the approach is implemented by controlling the exit criterion of the ATSP model. Comparative studies show that assignment strategies using both reviewers' and authors' preference matrices perform significantly better than random assignment. Notably, performance gains are minimized when these matrices are merged, suggesting a conflict in preferences between high-level and low-level students regarding "harmonious development."

Introduction

In educational settings, software peer review (or "peer code review") is a powerful pedagogical tool. However, many existing systems, such as the authors' EduPCR, traditionally rely on random assignment. While fair, random assignment fails to leverage the varying competence levels of students to maximize learning outcomes. The challenge lies in the Reviewer Assignment Problem (RAP): determining who should review whom to balance feedback quality with educational growth, while avoiding "mutual admiration societies" where students simply praise each other's work.

Methodology

The authors propose a multi-step optimization approach integrated into the standard peer review lifecycle. This lifecycle involves teachers setting assignments, students submitting manuscripts, and a structured review-revision loop.

The workflow of the EduPCR system, illustrating the roles of teachers, web servers, and students.

The optimization process follows five key steps:

  1. Competence Ranking: Students are clustered into High, Middle, and Low levels based on historical programming scores using hierarchical clustering.
  2. Preference Mapping: A questionnaire captured how students prefer to be assigned. Weights were calculated for various scenarios (e.g., a Low-level student reviewing a High-level student) to create a Preference Matrix.
  3. Rule Matrix Construction: This matrix defines the "distance" (cost) of an assignment; a smaller distance represents a higher priority for that specific match.
  4. ATSP Modeling: The core optimization uses the Asymmetric Traveling Salesman Problem (ATSP) framework. Unlike standard assignment models, ATSP includes subtour elimination constraints.
  5. Exit Criterion Control: To prevent cliques, the solver (IBM ILOG CPLEX) is programmed to stop only when every "subtour" (the loop of reviewers and authors) is at least three students long.
An example of a preference matrix showing the weighted costs of assignments between different competence levels.

Key Insights and Results

The study utilized survey data from 94 students and experimental data from 23 freshmen in a "C Programming" course. The model was evaluated using a relative ratio ($C = T_{random} / T_{optimized}$), where $C > 1.0$ indicates the optimization outperformed random assignment.

  • Reviewer Preference ($C_1$): Achieved an average ratio of 1.39.
  • Author Preference ($C_2$): Achieved the highest average ratio of 1.70.
  • Merged Preference ($C_3$): The ratio dropped to 1.11.

A striking finding was the "Harmonious Development" paradox: Low-level students are eager to interact with High-level students to improve, while High-level students rationally prefer to interact with their peers. When these conflicting preferences are merged, the optimization performance drops, suggesting that "what the reviewer wants" and "what the author wants" are often at odds in a classroom setting.

Conclusion

By framing reviewer assignment as an ATSP problem, this research demonstrates that structured preferences can significantly optimize the "intellectual distance" covered in a peer review cycle compared to random assignment. Future work will explore specialized rule matrices designed to boost the performance of specific student groups and investigate the long-term "orientation effect" of these strategies on organizational competence growth.

Summary updated

Abstract

Optimized reviewer assignment can effectively utilize limited intellectual resources and significantly assure review quality in various scenarios such as paper selection in conference or journal, proposal selection in funding agencies and so on. However, little research on reviewer assignment of software peer review has been found. In this study, an optimization approach is proposed based on students' preference matrix and the model of asymmetric traveling salesman problem (ATSP). Due to the most critical role of rule matrix in this approach, we conduct a questionnaire to obtain students' preference matrices and convert them to rule matrices. With the help of software ILOG CPLEX, the approach is accomplished by controlling the exit criterion of ATSP model. The comparative study shows that the assignment strategies with both reviewers' preference matrix and authors' preference matrix get better performance than the random assignment. Especially, it is found that the performance is just a little better than that of random assignment when the reviewers' and authors' preference matrices are merged. In other words, the majority of students have a strong wish of harmonious development even though high-level students are not willing to do that.
BibTeX
@misc{wang2014solving,
  author = {Yanqing Wang and Yu Jiang and Xiaolei Wang and Siyu Zhang and Yaowen Liang},
  title = {{Solving reviewer assignment problem in software peer review: An approach based on preference matrix and asymmetric TSP model}},
  year = {2014},
  eprint = {1401.4516},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1401.4516},
}
Figure