Publishing your scholarship: a survey of pearls from top reviewers

Joshua Jauregui, Anthony R. Artino, Jonathan S. Ilgen, Gail M. Sullivan, Sandrijn M. van SchaikMedical 2022 · DOI

TL;DR

This study surveyed top peer reviewers in health professions education to understand their feedback strategies. It reveals that while most reviewers don't use templates, they consistently focus on the "what" (research question/methods), "how" (presentation/writing), and "why" (relevance/impact) of a manuscript, adapting their focus based on the paper's overall quality.

What Problem Does This Paper Address?

Publishing high-quality research is crucial, and peer review is a cornerstone of this process. While much has been written about improving peer review, the actual perspectives of experienced peer reviewers themselves are often missing from the literature. This gap makes it difficult for authors to understand what reviewers are looking for and for new reviewers to develop effective strategies. The authors aim to bridge this gap by directly asking experienced reviewers how they approach their task, particularly how they differentiate between good and bad papers and what aspects they prioritize in their feedback.

Methods

The researchers employed a cross-sectional survey design, targeting "Top Reviewers" from the Journal of Graduate Medical Education (JGME) between 2017 and 2019. These reviewers were identified based on performing at least four reviews per year and receiving high internal quality ratings. An 11-item survey was administered, comprising demographic questions and eight open-ended, free-text questions focused on reviewers' approaches and common feedback areas.

Data analysis involved calculating descriptive statistics for demographic and quantitative survey items. For the open-ended responses, a thematic analysis was conducted. The researchers used a constructivist approach, meaning they acknowledged that meaning is co-created through the interaction between the researchers and the participants. All authors read the responses, met to discuss findings, and then divided into groups to independently code the data. Coding was reconciled, and categories were refined into major themes.

Datasets

Not discussed in the paper. The study surveyed individuals and did not use any pre-existing datasets for training or evaluation.

Key Insights and Ideas

  • The "What, How, Why" Framework: Reviewers consistently categorize their feedback around three core elements: the "what" (research question and methodology), the "how" (presentation and writing), and the "why" (relevance and impact). This provides a structured way to think about manuscript quality.
  • Quality Dictates Focus: The emphasis placed on each of these elements varies significantly based on the reviewer's initial assessment of the paper's quality.
  • "Good" Papers: Optimizing the "How" and "Why": For papers deemed "really good," reviewers tend to accept the fundamental "what" and focus their feedback on refining the presentation, writing, and articulating the broader relevance and impact.
  • "Bad" Papers: Big Picture First: When a paper has significant flaws, reviewers prioritize "big picture" feedback, such as the study's overall value and impact, before delving into minor details. They often adopt a more direct and concise approach.
  • Empathy and Efficiency: Reviewers sometimes adopt an empathetic tone, especially for weaker papers, and may prioritize feedback areas for efficiency.
  • Templates are Rare, Strategies are Common: Surprisingly, very few top reviewers reported using "stock phrases" or formal templates. Instead, they share common strategies for differentiating paper quality.

Key Results

  • Response Rate: Of 62 eligible top reviewers, 44 (71%) responded to the survey.
  • Reviewer Experience: On average, respondents had reviewed for health professions education (HPE) journals for 10.4 years (SD = 8.5 years).
  • Institutional Affiliation: The vast majority (88.6%) worked at academic institutions.
  • Template Use: Only 18.2% reported using "stock phrases," and 15.9% reported using a reviewer template.
  • Thematic Analysis: The thematic analysis identified the "what," "how," and "why" as the primary categories of reviewer concern.
  • For "really good" papers, reviewers focused on optimizing the "how" (writing, organization, flow) and "why" (generalizability, transferability, impact).
  • For "really bad" papers, reviewers focused on the "big picture" (value, impact) and overall strengths/weaknesses, often with an empathetic tone.

Limitations and Future Work

The authors acknowledge several limitations:

  • Single Journal Focus: The findings are based on reviewers from a single journal (JGME), which may limit generalizability to other journals or fields.
  • "Top Reviewer" Skew: The study focused on "top reviewers," a select group who may not represent the broader reviewer population.
  • Survey Development: The initial survey was developed by author consensus, which could have influenced the scope of questions.
  • Qualitative Data Depth: Open-ended survey questions, while rich, do not allow for the follow-up necessary to probe for deeper insights, as might be possible in interviews.

Future work suggested includes:

  • Studying reviewers who consistently produce high-quality reviews across different journals.
  • Exploring the impact of high-quality reviews on authors' work and manuscript revisions.
  • Investigating whether specific review elements are more beneficial to authors.
  • Considering the inclusion of reviewer training in graduate HPE programs to address the apparent gap in formal training.

Summary updated

Abstract

Experts have described ways to improve peer review quality. Perspectives <i>from</i> expert reviewers are largely absent in the health professions education literature. To gather guidance from expert reviewers, to aid authors striving to publish and reviewers aiming to perform their task effectively. This study surveyed <i>the Journal of Graduate Medical Education</i> (JGME) 'Top Reviewers' from 2017, 2018, and 2019. 'Top Reviewers' perform four or more reviews per year, with high average ratings. Top reviewers were sent an 11-item survey in February 2020. The survey included three demographic questions and eight open-ended, free-text questions about the concepts reviewers most often target in their reviews. We calculated descriptive statistics and performed a thematic analysis of open-ended responses. Of 62 eligible top reviewers, 44 (71%) responded to the survey. Only eight (18.2%) and seven (15.9%) respondents reported having 'stock phrases' or a reviewer template used for reviewer feedback to authors, respectively. The <i>what</i> (research question, methods), <i>how</i> (presentation, writing), and <i>why</i> (relevance, impact) were the resulting themes summarizing how reviewers categorized and responded to common problems. For 'really good papers' reviewers found the <i>what</i> acceptable and focused on <i>how</i> and <i>why</i>. For 'really bad' papers, reviewers focused on big picture feedback, such as the value of the study. Top reviewers from a single health professions education journal appear to have similar approaches to conducting reviews. While most do not use stock phrases or templates, they share similar strategies to differentiate 'good' vs. 'bad' papers through the <i>what, why</i>, and <i>how</i> of a manuscript.
BibTeX
@inproceedings{jauregui2022publishing,
  author = {Joshua Jauregui and Anthony R. Artino and Jonathan S. Ilgen and Gail M. Sullivan and Sandrijn M. van Schaik},
  title = {{Publishing your scholarship: a survey of pearls from top reviewers}},
  booktitle = {Medical},
  year = {2022},
  doi = {10.1080/10872981.2021.2016561},
}
Figure