Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process

Jing Yang

TL;DR

The AI/ML community is grappling with an explosion of submissions, leading many conferences to adopt more transparent peer review models. This paper analyzes the effectiveness of "fully open," "partially open," and "closed" review systems using data from the Paper Copilot platform, advocating for a more transparent, open, and regulated approach to foster community engagement and advance the field.

What Problem Does This Paper Address?

The exponential growth in submissions to top-tier AI and Machine Learning (AI/ML) conferences has put immense pressure on traditional peer review processes, challenging their fairness, efficiency, and quality. In response, many conferences are experimenting with open peer review (OPR) models, ranging from full public visibility of reviews and discussions to releasing them only after decisions. However, the optimal approach remains debated, with concerns about potential biases, reviewer candor, and the balance between transparency and intellectual property protection. This paper aims to shed light on community interest and the effectiveness of different models to inform the development of a more robust and transparent peer review system for AI/ML.

Methods

The authors leverage data from Paper Copilot, a website they launched to aggregate and analyze AI/ML conference data. This platform collects information from official conference websites, review platforms (like OpenReview), and community submissions via Google Forms. They employ custom bots to retrieve publicly available review metrics (ratings, confidence, comments) and analyze website traffic and user demographics using Google Analytics and Matomo. The analysis categorizes conferences into three review disclosure modes: Fully Open (all content public in real-time), Partially Open (reviews public after decisions), and Fully Closed (reviews remain private).

Datasets

The primary dataset is derived from Paper Copilot, which aggregates data from AI/ML conferences over a 10-year period (2015-2024) across 24 venues and 9 subfields. This includes:

  • Publicly available conference data: Sourced via APIs and web scraping from conference websites and platforms like OpenReview.
  • Community submissions: Anonymized review information voluntarily submitted by authors via Google Forms, particularly for closed-review venues.
  • Google Analytics data: Used to track website traffic, user demographics, and engagement patterns for Paper Copilot.

Key Insights and Ideas

  • Strong Global Interest in Transparency: The Paper Copilot platform has attracted over 200,000 active users from 177 countries, with a significant portion being early-career researchers, indicating a widespread desire for transparency in the peer review process.
  • Fully Open Reviews Drive Engagement: Conferences with "Fully Open" review models (like ICLR) show substantially higher community engagement (views, active users, session duration) compared to "Partially Open" or "Fully Closed" venues.
  • Transparency Doesn't Necessarily Mean More Confidence: While engagement is higher with open reviews, average confidence scores among reviewers remain consistent across different review models, suggesting that transparency alone doesn't inherently boost reviewer confidence.
  • Early-Career Researchers are Key Drivers: Younger researchers (18-24 age group) are the most active users of transparency-focused platforms, indicating their strong interest in understanding and participating in the review process.
  • Search Engines Reflect Community Needs: High search engine rankings for queries related to conference statistics and accepted papers demonstrate that researchers actively seek open and accessible information about review outcomes.
  • Challenges Remain for Closed Systems: Closed review systems face issues with reviewer experience, potential LLM misuse, and inconsistencies in author information, highlighting the need for greater accountability.

Key Results

  • Paper Copilot has attracted approximately 200,000 active users globally, with a peak of 15,000 unique visitors daily.
  • Conferences with Fully Open review models (e.g., ICLR) receive nearly four times more views and six times more active users than Partially Open (e.g., NeurIPS) or Fully Closed venues.
  • Average engagement time for users on Paper Copilot is around 3 minutes and 50 seconds for Fully Open venues, significantly higher than for Fully Closed venues.
  • Click-Through Rates (CTR) from Google Search to conference statistics pages are consistently high, ranging from 66.08% to 86.49%, regardless of the venue's transparency level.
  • Review confidence scores are consistently around 3.5 ± 0.5 across different review models and venues.
  • Discussion activity (number of replies) is significantly higher in Fully Open venues like ICLR compared to Partially Open venues like NeurIPS.

Limitations and Future Work

  • Data Bias: The reliance on voluntary submissions for closed-review venues may introduce biases, as not all authors may participate, potentially skewing the representation of community sentiment.
  • Rapid Evolution: The AI/ML community and its review norms are evolving rapidly, meaning current findings might become outdated quickly.
  • Limited Scope of Analysis: While the paper analyzes various aspects of transparency, it acknowledges that a deeper dive into specific mechanisms and the impact of LLMs on review quality could be beneficial.
  • Future Work: The authors suggest expanding data collection, refining demographic analyses, and further exploring peer review mechanisms to better understand and shape the future of AI/ML peer review. They also highlight the need for future studies to systematically analyze anecdotal claims about reviewer understanding of submissions.

Summary updated

Abstract

The rapid growth of submissions to top-tier Artificial Intelligence (AI) and Machine Learning (ML) conferences has prompted many venues to transition from closed to open review platforms. Some have fully embraced open peer reviews, allowing public visibility throughout the process, while others adopt hybrid approaches, such as releasing reviews only after final decisions or keeping reviews private despite using open peer review systems. In this work, we analyze the strengths and limitations of these models, highlighting the growing community interest in transparent peer review. To support this discussion, we examine insights from Paper Copilot ([papercopilot.com](https://papercopilot.com/)), a website launched two years ago to aggregate and analyze AI / ML conference data while engaging a global audience. The site has attracted over 200,000 early-career researchers, particularly those aged 18–34 from 177 countries, many of whom are actively engaged in the peer review process. \textit{Drawing on our findings, this position paper advocates for a more transparent, open, and well-regulated peer review aiming to foster greater community involvement and propel advancements in the field.
BibTeX
@inproceedings{yang2025position,
  author = {Jing Yang},
  title = {{Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=gnyqRarPzW},
}
Figure