TL;DR
The AI/ML community is grappling with an explosion of submissions, leading many conferences to adopt more transparent peer review models. This paper analyzes the effectiveness of "fully open," "partially open," and "closed" review systems using data from the Paper Copilot platform, advocating for a more transparent, open, and regulated approach to foster community engagement and advance the field.
What Problem Does This Paper Address?
The exponential growth in submissions to top-tier AI and Machine Learning (AI/ML) conferences has put immense pressure on traditional peer review processes, challenging their fairness, efficiency, and quality. In response, many conferences are experimenting with open peer review (OPR) models, ranging from full public visibility of reviews and discussions to releasing them only after decisions. However, the optimal approach remains debated, with concerns about potential biases, reviewer candor, and the balance between transparency and intellectual property protection. This paper aims to shed light on community interest and the effectiveness of different models to inform the development of a more robust and transparent peer review system for AI/ML.
Methods
The authors leverage data from Paper Copilot, a website they launched to aggregate and analyze AI/ML conference data. This platform collects information from official conference websites, review platforms (like OpenReview), and community submissions via Google Forms. They employ custom bots to retrieve publicly available review metrics (ratings, confidence, comments) and analyze website traffic and user demographics using Google Analytics and Matomo. The analysis categorizes conferences into three review disclosure modes: Fully Open (all content public in real-time), Partially Open (reviews public after decisions), and Fully Closed (reviews remain private).
Datasets
The primary dataset is derived from Paper Copilot, which aggregates data from AI/ML conferences over a 10-year period (2015-2024) across 24 venues and 9 subfields. This includes:
- Publicly available conference data: Sourced via APIs and web scraping from conference websites and platforms like OpenReview.
- Community submissions: Anonymized review information voluntarily submitted by authors via Google Forms, particularly for closed-review venues.
- Google Analytics data: Used to track website traffic, user demographics, and engagement patterns for Paper Copilot.
Key Insights and Ideas
- Strong Global Interest in Transparency: The Paper Copilot platform has attracted over 200,000 active users from 177 countries, with a significant portion being early-career researchers, indicating a widespread desire for transparency in the peer review process.
- Fully Open Reviews Drive Engagement: Conferences with "Fully Open" review models (like ICLR) show substantially higher community engagement (views, active users, session duration) compared to "Partially Open" or "Fully Closed" venues.
- Transparency Doesn't Necessarily Mean More Confidence: While engagement is higher with open reviews, average confidence scores among reviewers remain consistent across different review models, suggesting that transparency alone doesn't inherently boost reviewer confidence.
- Early-Career Researchers are Key Drivers: Younger researchers (18-24 age group) are the most active users of transparency-focused platforms, indicating their strong interest in understanding and participating in the review process.
- Search Engines Reflect Community Needs: High search engine rankings for queries related to conference statistics and accepted papers demonstrate that researchers actively seek open and accessible information about review outcomes.
- Challenges Remain for Closed Systems: Closed review systems face issues with reviewer experience, potential LLM misuse, and inconsistencies in author information, highlighting the need for greater accountability.
Key Results
- Paper Copilot has attracted approximately 200,000 active users globally, with a peak of 15,000 unique visitors daily.
- Conferences with Fully Open review models (e.g., ICLR) receive nearly four times more views and six times more active users than Partially Open (e.g., NeurIPS) or Fully Closed venues.
- Average engagement time for users on Paper Copilot is around 3 minutes and 50 seconds for Fully Open venues, significantly higher than for Fully Closed venues.
- Click-Through Rates (CTR) from Google Search to conference statistics pages are consistently high, ranging from 66.08% to 86.49%, regardless of the venue's transparency level.
- Review confidence scores are consistently around 3.5 ± 0.5 across different review models and venues.
- Discussion activity (number of replies) is significantly higher in Fully Open venues like ICLR compared to Partially Open venues like NeurIPS.
Limitations and Future Work
- Data Bias: The reliance on voluntary submissions for closed-review venues may introduce biases, as not all authors may participate, potentially skewing the representation of community sentiment.
- Rapid Evolution: The AI/ML community and its review norms are evolving rapidly, meaning current findings might become outdated quickly.
- Limited Scope of Analysis: While the paper analyzes various aspects of transparency, it acknowledges that a deeper dive into specific mechanisms and the impact of LLMs on review quality could be beneficial.
- Future Work: The authors suggest expanding data collection, refining demographic analyses, and further exploring peer review mechanisms to better understand and shape the future of AI/ML peer review. They also highlight the need for future studies to systematically analyze anecdotal claims about reviewer understanding of submissions.
Summary updated