TL;DR
This paper demystifies the "black box" of academic publishing by interviewing editors from the Journal of Engineering Education (JEE). It reveals the behind-the-scenes mechanics of how reviewers are chosen, why papers get rejected, and how editors reconcile conflicting reviews to make a final decision.
What Problem Does This Paper Address?
For many researchers—especially graduate students and early-career faculty—the peer review process feels like a cryptic, high-stakes "black box." You submit a manuscript, wait months, and receive a decision that can feel arbitrary or overly harsh.
This opacity is particularly challenging in Engineering Education Research (EER), an interdisciplinary field where reviewers may come from vastly different methodological backgrounds (e.g., qualitative vs. quantitative). The authors aim to "illuminate" this process by providing transparency into the editorial mindset, helping scholars better navigate the path to publication.
Methods
The researchers conducted a qualitative study using semi-structured interviews. They interviewed six editors from JEE, including Associate Editors, Senior Associate Editors, and Deputy Editors.
- Approach: Interviews lasted approximately one hour and followed a specific protocol but allowed for follow-up questions.
- Analysis: The interviews were recorded, transcribed, and analyzed using thematic open-coding. The research team held weekly meetings to reach "inter-rater reliability," ensuring that their interpretation of the editors' comments was consistent and accurate.
Datasets
As a qualitative study, there is no traditional numerical dataset. The "data" consists of the transcripts from the six interviews with JEE editors conducted in Spring 2019.
Key Insights and Ideas
- The Editor Hierarchy: The process isn't a monolith. Deputy Editors perform the initial "gatekeeping" (desk rejects), while Associate Editors do the heavy lifting of recruiting reviewers and synthesizing their feedback. The final decision rests with the Editor or Deputy Editor based on the Associate Editor’s recommendation.
- The "Expert Blind Spot": Editors noted that authors are often so close to their work that they fail to explain it clearly to an outside audience. This "blind spot" is a leading cause of reviewer confusion.
- Finding Reviewers is a Craft: Editors don't just use automated systems. They hunt through a manuscript’s reference list, search Google Scholar, and tap into personal networks. Interestingly, they find author-suggested reviewers helpful but will always include at least one reviewer the author didn't suggest to ensure impartiality.
- Reconciling Conflict: A key job of the Associate Editor is to "reconcile conflicting reviews." They don't just count votes; they evaluate which reviewer’s criticism is "legitimate" based on the field’s standards and the specific methodology used.
- The "Major Revision" Safety Net: Editors often use the "Major Revision" decision as a way to be supportive. It gives authors a chance to clarify if a study was "poorly designed" or just "poorly written."
Key Results
- High Stakes: The Journal of Engineering Education (JEE) is highly competitive, maintaining a rejection rate of approximately 90% over the last five years.
- Four Pillars of Desk Rejection: Papers are typically rejected immediately if they:
- Are entirely out of scope (e.g., technical engineering research with no education component).
- Lack data (e.g., describing an intervention without measuring its impact).
- Have fundamental flaws in design/methodology that cannot be "saved."
- Are poorly organized (missing theoretical frameworks or clear research questions).
- Reviewer Tone Matters: Editors expressed a need for more "supportive" feedback. They explicitly advised reviewers that the first round of review is not the time for detailed proofreading or typo-hunting, as those sections might be deleted in revisions anyway.
Limitations and Future Work
The authors acknowledge several limitations:
- Small Sample Size: The study only interviewed six editors.
- Single-Journal Focus: The findings are specific to JEE and may not generalize to all journals in the field.
- Temporal Snapshot: The interviews were conducted in 2019; editorial boards and journal policies evolve over time.
Future Work: The authors are currently working on developing formal peer review guidelines to help reviewers reflect on their process and maintain a constructive, supportive tone. They also encourage the community to create more "space for dialogue" and for new journals to emerge to fill specific niches that JEE might not cover.
Summary updated