AutoRef: Generating Refinements of Reviews Given Guidelines

Soham Chitnis, Manasi Patwardhan, Ashwin Srinivasan, Tanmay Tulsidas Verlekar, Lovekesh Vig, Gautam ShroffACL 2024

TL;DR

This paper introduces AutoRef, an AI system designed to iteratively refine research paper reviews based on predefined guidelines. By transforming review requirements into a series of "yes/no" questions, AutoRef uses LLM-powered agents to assess and improve existing reviews, aiming to produce more consistent and compliant evaluations.

What Problem Does This Paper Address?

The peer-review process for scholarly articles, while crucial, faces challenges due to the sheer volume of submissions and the subjective nature of human reviews. This can lead to inconsistent feedback, with referees being either too lenient or overly strict, neither of which is ideal. Researchers aim to develop an AI assistant that can help generate or refine reviews to be more compliant with established guidelines, ensuring a more balanced and objective evaluation. This addresses the need for a more systematic and less error-prone review process, benefiting referees, meta-reviewers, and authors alike.

Methods

AutoRef employs an iterative refinement process. The core idea is to translate review guidelines into a structured set of "yes/no" questions. This forms a "structured questionnaire." For a given paper and review, AutoRef uses a "Reviewer Agent" (an LLM) to extract answers to these questions and generate a "structured review." This structured review is then fed back, along with the paper and the structured questionnaire, to a "Feedback Agent" (another LLM) which provides feedback on the review's quality. This feedback is used by the Reviewer Agent to refine the structured review in a loop. The process continues for a set number of iterations, aiming to improve the review's compliance with the guidelines. The system also includes functions to translate between natural language reviews and structured reviews.

Datasets

The study uses data from the International Conference on Learning Representations (ICLR) 2023. This includes:

  • Guidelines for reviewers: The official guidelines provided by ICLR.
  • Papers: Submitted research papers.
  • Human-referee reviews: Existing reviews from human referees.

For their experiments, they created two datasets:

  • Dataset D: A stratified selection of 300 papers from ICLR, with 140 accepted and 160 rejected. This dataset is used for experiments starting with an automatically generated review.
  • Dataset E: A subset of reviews categorized as potentially erroneous (e.g., accepted papers with negative reviews, or rejected papers with positive reviews). This dataset is used for experiments where AutoRef refines existing human reviews.

Key Insights and Ideas

  • Iterative Refinement: The core innovation is the iterative nature of AutoRef, where a review is progressively improved by repeatedly assessing it against structured questions and using LLM feedback.
  • Structured Questionnaires: Translating abstract guidelines into concrete "yes/no" questions is a key mechanism for making the review process more objective and measurable.
  • Stricter Reviews: The system tends to generate stricter reviews over iterations, pushing decisions more towards rejection. This is a valuable outcome for identifying weaknesses in papers.
  • LLM Agents for Review Tasks: The paper demonstrates the utility of LLMs not just for generating text, but for performing specific analytical tasks within the review process (extraction, assessment, feedback).

Key Results

  • Improved Review Scores: In experiments starting without a prior review, AutoRef showed a progressive increase in review scores over iterations, indicating improvement in compliance with guidelines. The average increase in score decreased with each iteration, suggesting convergence.
  • Refining Human Reviews: When refining existing human reviews, AutoRef also led to score improvements, though the number of samples showing improvement decreased over iterations.
  • Stricter Decision Tendency: AutoRef generated reviews showed a decrease in "Accept" decisions and an increase in "Reject" decisions over iterations, indicating a trend towards stricter evaluations. This also led to a reduction in True Positives (TPs) when trying to reduce False Positives (FPs).
  • Precision vs. Recall: Compared to human reviews, AutoRef generated reviews had lower precision for the "Accept" class but comparable recall. This suggests that while AutoRef is good at identifying papers that might be acceptable, it's more prone to incorrectly rejecting papers when aiming for stricter criteria.

Limitations and Future Work

  • Manual Task Ordering and Questionnaires: The current implementation relies on manually defined task orderings and structured questionnaires, which is a bottleneck. Automating this process would be a significant improvement.
  • LLM Hallucinations: The Reviewer Agent, being an LLM, can potentially "hallucinate" decisions, especially when the final decision is based on the outcomes of parent tasks. More deterministic algorithms could mitigate this.
  • Limited Scope of Questions: The current "yes/no" questions are described as fairly shallow and might not capture all nuances of a review. Developing more in-depth and sophisticated questions is a direction for future work.
  • Computational Cost: The paper notes that the method might be computationally expensive, and further analysis is needed.
  • Flexibility: The system might lack the flexibility of post-hoc counterfactual generation methods, requiring retraining for different notions of cost or actionability.
  • Few-Shot Learning: The paper did not utilize advanced LLM features like few-shot learning or generating multiple reviews, which could enhance performance.
  • Data Memorization: There's a possibility that the LLM might have memorized parts of the training data, making it less effective for showcasing novel capabilities.

Summary updated

BibTeX
@inproceedings{chitnis2024autoref,
  author = {Soham Chitnis and Manasi Patwardhan and Ashwin Srinivasan and Tanmay Tulsidas Verlekar and Lovekesh Vig and Gautam Shroff},
  title = {{AutoRef: Generating Refinements of Reviews Given Guidelines}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure