KID-Review: Knowledge-Guided Scientific Review Generation with Oracle Pre-training

Weizhe Yuan, Pengfei Liu

The exponential growth of scientific literature has placed an immense burden on the peer review process. Reviewers are often overwhelmed, leading to delays and potentially inconsistent feedback. To address this, researchers from Carnegie Mellon University have developed KID-Review, an end-to-end framework designed to generate constructive, first-pass scientific reviews. By integrating external "world knowledge" with internal "temporary knowledge" and employing a novel "oracle" training strategy, KID-Review moves beyond simple text summarization toward a more nuanced critique of research papers.

The Challenge of Automated Peer Review

Generating a scientific review is significantly more complex than standard document summarization. A high-quality review requires a model to:

  1. Synthesize Internal Logic: Understand the specific contributions, methods, and results presented within the paper (working memory).
  2. Contextualize Externally: Evaluate how the paper fits into the broader research landscape, including its relationship to existing literature and baselines (long-term memory).
  3. Overcome Training Sparsity: Map a lengthy, technical document to a concise, critical review—a "needle-in-a-haystack" problem where standard training often fails to find optimal solutions due to input noise.

Dual Knowledge Representation

To mimic human cognitive processes, KID-Review utilizes two distinct types of knowledge graphs to inform the generation process.

The KID-Review framework utilizes a concept graph (top) to represent internal paper logic and a citation graph (bottom) to capture the broader research context.
  • World Knowledge (Citation Graph): This represents the "long-term memory" of a scientific field. Using the S2ORC dataset, the authors constructed a graph of paper relationships. Paper embeddings, learned via the LINE algorithm, allow the model to assess a paper's integration and relevance within the existing literature.
  • Temporary Knowledge (Concept Graph): This represents "working memory." The system extracts entities (e.g., "BERT," "F1 score") and their relationships (e.g., "used-for," "evaluate-for") directly from the paper under review. This graph captures the internal structure and technical claims of the work.

System Architecture

The KID-Review architecture is built upon a modified BART model, specifically designed to ingest and process these diverse knowledge sources.

Architecture of the KID-Review model, featuring a BART-based backbone augmented with Graph Attention Networks (GAT) and citation embeddings.

As illustrated in the architecture diagram, citation embeddings are concatenated with the BART encoder's output. Simultaneously, the concept graph is processed through a Graph Attention Network (GAT). To integrate this structured information, the BART decoder is enhanced with an additional cross-attention layer that specifically attends to the graph-based concept representations. This allows the model to ground its generated text in both the paper's specific technical details and its broader academic context.

Oracle Pre-training Strategy

To mitigate the difficulties of training on long, noisy technical papers, the authors introduced Oracle Pre-training. Instead of training directly on full papers or abstracts, the model is first "warmed up" using "oracle" sentences. These are sentences extracted from the paper that exhibit the highest lexical overlap with actual human reviews. By training on these high-signal sentences first, the model learns to identify "review-worthy" content before being tasked with processing the full complexity of the document. This strategy provides a smoother optimization path and improves the model's ability to generalize during inference.

Key Results and Insights

The researchers evaluated KID-Review using the ASAP-Review dataset (comprising ICLR and NeurIPS papers) and the S2ORC corpus. Key findings include:

  • Improved Recommendation Accuracy: The inclusion of citation graphs improved the model's ability to predict acceptance or rejection, outperforming vanilla BART by nearly 10% in certain tests.
  • Enhanced Constructiveness: In double-blind human evaluations, reviews generated by the full KID-Review system were preferred over vanilla baselines 57% of the time.
  • Specific and Inquisitive Style: The concept-guided model was more likely to provide detailed feedback, frequently using inquisitive phrases and suggesting specific comparisons or baselines.

Conclusion and Future Directions

KID-Review demonstrates that explicitly modeling different types of knowledge and using targeted pre-training can significantly improve the quality of automated scientific reviews. However, challenges remain. The system can still suffer from factual hallucinations and struggles to assess subjective qualities like writing clarity. Future work will focus on more fine-grained citation modeling and improving the factuality of the generated critiques.

Summary updated

Abstract

The surge in the number of scientific submissions has brought challenges to the work of peer review. In this paper, as a first step, we explore the possibility of designing an automated system, which is not meant to replace humans, but rather providing a first-pass draft for a machine-assisted human review process. Specifically, we present an end-to-end knowledge-guided review generation framework for scientific papers grounded in cognitive psychology research that a better understanding of text requires different types of knowledge. In practice, we found that this seemingly intuitive idea suffered from training difficulties. In order to solve this problem, we put forward an oracle pre-training strategy, which can not only make the Kid-Review better educated but also make the generated review cover more aspects. Experimentally, we perform a comprehensive evaluation (human and automatic) from different perspectives. Empirical results have shown the effectiveness of different types of knowledge as well as oracle pre-training. We make all code, relevant dataset available: https://github.com/Anonymous4nlp233/KIDReview as well as the Kid-Review system: http://nlpeer.reviews.
BibTeX
@inproceedings{yuan2022kid,
  author = {Weizhe Yuan and Pengfei Liu},
  title = {{KID-Review: Knowledge-Guided Scientific Review Generation with Oracle Pre-training}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21418},
}
Figure