Abstract
Automatically validating a research artefact is one of the frontiers in Artificial Intelligence (AI) that directly brings it close to competing with human intellect and intuition. Although criticised sometimes, the existing peer review system still stands as the benchmark of research validation. The present-day peer review process is not straightforward and demands profound domain knowledge, expertise, and intelligence of human reviewer(s), which is somewhat elusive with the current state of AI. However, the peer review texts, which contains rich sentiment information of the reviewer, reflecting his/her overall attitude towards the research in the paper, could be a valuable entity to predict the acceptance or rejection of the manuscript under consideration. Here in this work, we investigate the role of reviewer sentiment embedded within peer review texts to predict the peer review outcome. Our proposed deep neural architecture takes into account three channels of information: the paper, the corresponding reviews, and review’s polarity to predict the overall recommendation score as well as the final decision. We achieve significant performance improvement over the baselines (~29% error reduction) proposed in a recently released dataset of peer reviews.
Introduction
The academic peer review system is currently under immense strain. With submission volumes for major AI conferences like CVPR and AAAI skyrocketing into the thousands, program chairs and editors are overwhelmed. This leads to issues such as inconsistency, reviewer shortages, and missing data. While current AI cannot replace the deep domain expertise of a human reviewer, it can act as an "additional layer of confidence." DeepSentiPeer is a deep learning framework designed to assist conference editors by predicting peer review outcomes by analyzing the interplay between the original paper, the text of the reviews, and the underlying sentiment of the reviewers.
Methodology
The authors propose a multi-channel deep neural architecture called DeepSentiPeer. The pipeline consists of three main stages:
- Document Encoding: The model converts full-text sentences from both the research paper and the reviews into high-dimensional embeddings using the Universal Sentence Encoder (USE).
- Sentiment Encoding: To capture the "human touch," the authors use the VADER Sentiment Analyzer to generate a sentiment polarity score (positive, negative, neutral, and compound) for every sentence in the review.
- Feature Extraction & Fusion: Convolutional Neural Networks (CNNs) extract structural features from the paper and review representations. The model uses feature-level fusion for predicting the Accept/Reject decision and decision-level fusion for predicting the Recommendation Score.
Key Insights
A central finding of this research is that reviewer sentiment serves as a powerful proxy for quality. Even when a review is written in dry, scholarly language, the underlying sentiment (captured by VADER) has visible correlations with specific aspects like "Originality" and "Soundness." The interaction between the paper's content and the reviewer's sentiment is where the predictive power lies.
Results
The model was evaluated on two primary tasks using the PeerRead dataset:
- Task 1: Recommendation Score Prediction (Regression): DeepSentiPeer achieved a ~29% reduction in error (RMSE) on the ICLR 2017 dataset compared to state-of-the-art baselines. The "Paper+Review+Sentiment" configuration consistently outperformed models using only paper or review text.
- Task 2: Accept/Reject Decision (Classification): On the ICLR 2017 test set, the model reached an accuracy of 71.05%, significantly higher than the "Only Paper" baseline of 55.26%.
Furthermore, the model showed cross-domain robustness, achieving a relative error reduction of up to 14.5% when trained on Machine Learning papers (ICLR) and tested on Computational Linguistics papers (ACL).
Conclusion
DeepSentiPeer demonstrates that harnessing sentiment in review texts can significantly improve the prediction of peer review outcomes. While scholarly sentiment is nuanced and often neutral, the model provides a valuable "second opinion" for overworked editors. Future work involves developing sentiment encoders specifically tuned for academic prose and incorporating a "review reliability" component to identify high-quality versus low-effort reviews.
Summary updated