ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing

Lu Sun, Aaron Chan, Yun Seo Chang, Steven P. Dow

Abstract

The academic community is currently facing a "reviewer crisis" characterized by a skyrocketing volume of paper submissions and a shortage of experienced reviewers. Consequently, conferences are increasingly recruiting novice reviewers, such as graduate students, who often lack the formal training and domain expertise required to produce high-quality critiques. This paper introduces ReviewFlow, an AI-powered intelligent scaffolding system designed to assist novice researchers in navigating the complexities of the peer review process. By integrating real-time critical thinking prompts, in-situ citation context, and automated note-to-outline synthesis, ReviewFlow helps novices produce more comprehensive, expert-like reviews while significantly boosting their confidence.

Introduction

Peer review is the cornerstone of academic publishing, yet the process remains largely opaque to newcomers. Novice reviewers typically face three primary hurdles: a lack of domain-specific knowledge to evaluate novelty, difficulty in synthesizing scattered annotations into a coherent argument, and an unfamiliarity with the "unwritten rules" regarding the structure and tone of a professional review. While standard text editors provide a blank canvas, they offer no guidance during the critical reading and sense-making phases. ReviewFlow addresses these gaps by providing a structured environment that scaffolds the cognitive process of reviewing rather than simply automating the output.

The ReviewFlow System

ReviewFlow was developed following a formative study involving 10 novices and 10 experts to map the "expert workflow." The system is a web-based platform that integrates three core features:

  1. Contextual Cues: Utilizing GPT-4, the system generates section-specific questions (e.g., "Does the methodology justify the choice of participants?") to prompt critical reflection as the user reads the manuscript.
  2. In-situ Knowledge Support: To help novices judge novelty and context, the system pulls summaries of cited papers via the Semantic Scholar API and suggests relevant missing citations.
  3. Notes-to-Outline Synthesis: To lower the "blank page" hurdle, an LLM organizes the reviewer’s raw marginalia and notes into a structured draft categorized by strengths and weaknesses, which the user can then refine and expand.

Methodology

The researchers evaluated ReviewFlow through a within-subjects experiment with 16 novice reviewers. Each participant reviewed two different short papers from top-tier Human-Computer Interaction (HCI) conferences (e.g., CHI, CSCW). One review was conducted using ReviewFlow, while the other was performed in a baseline environment consisting of a standard text editor and a static rubric. The study aimed to measure the impact of scaffolding on review quality, the reviewer's cognitive process, and their self-efficacy.

Key Results

The evaluation yielded several statistically significant improvements for participants using ReviewFlow:

  • Improved Quality and Comprehensiveness: Expert evaluations indicated that ReviewFlow users produced significantly more comprehensive reviews (p=0.04) compared to the baseline. Reviewers were able to identify more specific strengths and weaknesses within the manuscripts.
  • Shift in Cognitive Effort: Participants using ReviewFlow spent 40% more time on reading and reflection (26.9 minutes vs. 19.2 minutes) than those in the baseline group. Despite this increased engagement, they reported that the final writing phase was more streamlined due to the automated synthesis of their notes.
  • Productive Distraction: While users noted that AI prompts were occasionally distracting, they characterized this as "cognitive forcing" that prevented passive skimming and encouraged deeper engagement with the text.
  • Increased Self-Efficacy: Novices reported a significant boost in their confidence to perform peer reviews, with self-efficacy scores rising from 3.92 to 4.92 on a 7-point scale.

Discussion and Future Work

ReviewFlow demonstrates the potential of "scaffolding over automation." By acting as a mentor rather than a ghostwriter, the system supports the human's cognitive process while maintaining the reviewer's agency. However, the study also highlighted areas for improvement. For instance, while reviews became more comprehensive, novices still struggled to provide "actionable" suggestions for authors. Future iterations may include critique style guides to help transform identified weaknesses into constructive feedback.

Additionally, the researchers identified concerns regarding AI bias and potential hallucinations, which could inadvertently influence a reviewer's judgment. Future work will also explore the concept of "fading"—investigating whether reviewers eventually internalize these expert habits and can maintain high performance without the system's assistance—and test the system's efficacy on full-length journal submissions.

Summary updated

BibTeX
@inproceedings{sun2024reviewflow,
  author = {Lu Sun and Aaron Chan and Yun Seo Chang and Steven P. Dow},
  title = {{ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing}},
  booktitle = {IUI},
  year = {2024},
  doi = {10.1145/3640543.3645159},
}
Figure