Abstract
Optimized reviewer assignment can effectively utilize limited intellectual resources and significantly assure review quality in various scenarios such as paper selection in conferences or journals, and proposal selection in funding agencies. However, little research on reviewer assignment for software peer review has been found. In this study, an optimization approach is proposed based on students' preference matrices and the model of the Asymmetric Traveling Salesman Problem (ATSP). Due to the critical role of the rule matrix in this approach, we conducted a questionnaire to obtain students' preference matrices and converted them to rule matrices. Using IBM ILOG CPLEX, the approach is implemented by controlling the exit criterion of the ATSP model. Comparative studies show that assignment strategies using both reviewers' and authors' preference matrices perform significantly better than random assignment. Notably, performance gains are minimized when these matrices are merged, suggesting a conflict in preferences between high-level and low-level students regarding "harmonious development."
Introduction
In educational settings, software peer review (or "peer code review") is a powerful pedagogical tool. However, many existing systems, such as the authors' EduPCR, traditionally rely on random assignment. While fair, random assignment fails to leverage the varying competence levels of students to maximize learning outcomes. The challenge lies in the Reviewer Assignment Problem (RAP): determining who should review whom to balance feedback quality with educational growth, while avoiding "mutual admiration societies" where students simply praise each other's work.
Methodology
The authors propose a multi-step optimization approach integrated into the standard peer review lifecycle. This lifecycle involves teachers setting assignments, students submitting manuscripts, and a structured review-revision loop.
The optimization process follows five key steps:
- Competence Ranking: Students are clustered into High, Middle, and Low levels based on historical programming scores using hierarchical clustering.
- Preference Mapping: A questionnaire captured how students prefer to be assigned. Weights were calculated for various scenarios (e.g., a Low-level student reviewing a High-level student) to create a Preference Matrix.
- Rule Matrix Construction: This matrix defines the "distance" (cost) of an assignment; a smaller distance represents a higher priority for that specific match.
- ATSP Modeling: The core optimization uses the Asymmetric Traveling Salesman Problem (ATSP) framework. Unlike standard assignment models, ATSP includes subtour elimination constraints.
- Exit Criterion Control: To prevent cliques, the solver (IBM ILOG CPLEX) is programmed to stop only when every "subtour" (the loop of reviewers and authors) is at least three students long.
Key Insights and Results
The study utilized survey data from 94 students and experimental data from 23 freshmen in a "C Programming" course. The model was evaluated using a relative ratio ($C = T_{random} / T_{optimized}$), where $C > 1.0$ indicates the optimization outperformed random assignment.
- Reviewer Preference ($C_1$): Achieved an average ratio of 1.39.
- Author Preference ($C_2$): Achieved the highest average ratio of 1.70.
- Merged Preference ($C_3$): The ratio dropped to 1.11.
A striking finding was the "Harmonious Development" paradox: Low-level students are eager to interact with High-level students to improve, while High-level students rationally prefer to interact with their peers. When these conflicting preferences are merged, the optimization performance drops, suggesting that "what the reviewer wants" and "what the author wants" are often at odds in a classroom setting.
Conclusion
By framing reviewer assignment as an ATSP problem, this research demonstrates that structured preferences can significantly optimize the "intellectual distance" covered in a peer review cycle compared to random assignment. Future work will explore specialized rule matrices designed to boost the performance of specific student groups and investigate the long-term "orientation effect" of these strategies on organizational competence growth.
Summary updated