SciCompanion: Graph-Grounded Reasoning for Structured Evaluation of Scientific Arguments

Joshua Alan Flashner, Adithya Kulkarni, Dawei ZhouEMNLP 2025

TL;DR

The scientific publication explosion is overwhelming peer review. SCICOMPANION tackles this by using a novel framework that combines large language models with dynamic knowledge graphs and reinforcement learning to provide structured, multi-hop reasoning for scientific evaluation. This allows smaller models to outperform larger ones and produces more accurate, interpretable reviews.

What Problem Does This Paper Address?

The sheer volume of scientific publications is creating an unsustainable workload for reviewers and researchers. Top conferences receive thousands of submissions annually, forcing reviewers to assess complex papers under tight deadlines with limited tools to identify relevant prior work. This is particularly challenging for early-career researchers navigating rapidly evolving fields. Existing AI systems, like standard retrieval-augmented generation (RAG) models, struggle with static retrieval, surface-level analysis, and lack the multi-hop reasoning needed for deep, contextual understanding, often leading to shallow or inaccurate assessments. The paper aims to bridge this gap by providing an intelligent, scalable, and trustworthy system for synthesizing, contextualizing, and evaluating scientific contributions.

Methods

SCICOMPANION employs a unique, iterative framework that integrates three key components:

  • Structured Graph Retrieval: Instead of static retrieval, SCICOMPANION builds a dynamic knowledge graph from recent publications, domain-specific databases, and curated metadata. This graph is iteratively refined through multi-hop reasoning.
  • Graph-Guided Reasoning: The system uses an LLM-based agent policy to dynamically construct contextual graphs, formulate hypotheses, and generate structured critiques. This process is underpinned by two core representations: a state graph (for accumulated structured knowledge) and a notebook (for free-form reasoning).
  • Reinforcement Learning for Alignment: To ensure reviews align with expert judgments, SCICOMPANION optimizes retrieval and graph refinement using Group Relative Policy Optimization (GRPO). This reinforcement learning approach allows the model to learn adaptive retrieval and critique strategies that generalize across different papers and review dimensions, promoting both accuracy and interpretability.

The framework iteratively retrieves information, refines the knowledge graph, and generates reasoning traces, ultimately producing structured assessments of feasibility, novelty, and impact.

Datasets

The paper evaluates SCICOMPANION on three diverse datasets:

  • ICLR (5,482 ML papers with reviews): Used for testing multi-dimensional critique (feasibility, novelty, impact).
  • ACL Soundness & Overall (3,219 CL papers): Used for labels related to methodological rigor and overall recommendation.
  • GoodReads (5,000 book descriptions with user ratings): Used to test cross-domain adaptability with loosely structured text and non-expert preferences.

Key Insights and Ideas

  • Structured Reasoning Trumps Scale: The paper demonstrates that smaller models (e.g., 7B parameters) using SCICOMPANION's structured, graph-guided reasoning can outperform larger, prompting-only models. This highlights the power of intelligent architecture and reasoning over raw model size.
  • Multi-hop Retrieval is Crucial: Increasing retrieval steps from one to three significantly improves evaluation accuracy (reduces RMSE), indicating the necessity of exploring deeper connections within the literature. However, gains plateau beyond three steps, suggesting an optimal trade-off between accuracy and computational cost.
  • Reinforcement Learning for Expert Alignment: GRPO is essential for aligning the model's outputs with expert review criteria, leading to more faithful and interpretable critiques than standard prompting or supervised fine-tuning.
  • Dynamic Graph Construction: The ability to dynamically build and refine knowledge graphs is key to capturing latent and implicit connections, which static retrieval methods miss.

Key Results

  • Reduced Evaluation Error: SCICOMPANION reduces evaluation error by over 30% compared to prompting-only baselines.
  • Improved RMSE: On the ICLR dataset, the 14B variant of SCICOMPANION achieved an RMSE of 0.95, outperforming fine-tuned 14B models and even surpassing PeerRead.
  • Enhanced Retrieval Accuracy: SCICOMPANION retrieves significantly more ground-truth references than RAG-based models, with the 14B variant retrieving 45.29% of actual references in an abstract-only setting, nearly doubling RAG's 27.18%.
  • Higher Point Match Rates: The system achieves higher strong and weak point match rates with peer reviews, indicating greater fidelity to expert evaluations. The 14B variant achieved a 70.9% strong match rate.
  • Smaller Models Compete: The 7B variant of SCICOMPANION consistently outperforms larger prompting baselines, showcasing the effectiveness of its architecture.

Limitations and Future Work

The authors acknowledge several limitations:

  • Corpus Dependence: Reliance on curated knowledge graphs and open-access corpora might limit coverage in underrepresented or rapidly evolving scientific domains, potentially degrading evaluation quality.
  • Supervised Data Requirement: GRPO optimization requires supervised review data, which may not be universally available.
  • Language Generalizability: The current evaluation focuses on English-language texts; multilingual capabilities are untested.
  • Deeper Critique Aspects: While point-matching metrics capture surface agreement, they don't fully reflect deeper aspects of critique quality like originality or epistemic humility.
  • Long-Term Impact: The long-term effects of automated review assistance on human decision-making and reviewer behavior are not yet evaluated.

Future work could explore expanding corpus coverage, developing methods for low-resource languages, and investigating the broader impact on the academic review ecosystem.

Summary updated

BibTeX
@inproceedings{flashner2025scicompanion,
  author = {Joshua Alan Flashner and Adithya Kulkarni and Dawei Zhou},
  title = {{SciCompanion: Graph-Grounded Reasoning for Structured Evaluation of Scientific Arguments}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure