SciPIP: An LLM-based Scientific Paper Idea Proposer

Wenxiao Wang, Lihui Gu, Liye Zhang, Yunxiang Luo, Yi Dai, Chen Shen, Liang Xie, Binbin Lin, Xiaofei He, Jieping YeSubmission to ICLR 2025 · Withdrawn

TL;DR

SciPIP (Scientific Paper Idea Proposer) is an LLM-based framework designed to autonomously propose novel and feasible research ideas from a user-supplied problem statement. By combining multi-level literature retrieval (semantics, entities, and citation co-occurrence) with a dual-path ideation pipeline (literature-grounded deduction plus LLM brainstorming), SciPIP discovers ideas that align with top-tier conference publications while generating hundreds of unexplored, high-novelty hypotheses.

What Problem Does This Paper Address?

Generating impactful research hypotheses requires balancing novelty (breaking new ground) with feasibility (grounding ideas in realistic methodologies). While LLMs show great promise in scientific ideation, existing automated idea proposers struggle with two main obstacles:

  1. Ineffective literature retrieval: Naive keyword searches often miss subtle interdisciplinary connections or pull in redundant, noisy papers that dilute the model's focus.
  2. The novelty–feasibility tradeoff: Unconstrained LLMs often hallucinate infeasible solutions, while purely literature-constrained systems tend to propose incremental tweaks or simple method transplants.

Methods

SciPIP structures the ideation process into three primary stages:

[User Background] ──► [SEC Retrieval & Clustering] ──► [Dual-Path Generation & Refinement] ──► [Final Ideas]
  1. Literature Database Construction: The authors parsed 48,895 AI/NLP papers from the last decade (ICLR, NeurIPS, ICML, ACL, NAACL, EMNLP). Using GLM-4 and Sentence-BERT, they extracted backgrounds, summaries, main/detailed ideas, entities, and core references (citations from Introduction and Method sections).
  2. SEC-Based Retrieval & Clustering:
  3. Semantics (Macro): Sentence-BERT embeddings retrieve the top-55 semantically similar paper backgrounds.
  4. Entities (Micro): Extracts entities and filters expansions using co-occurrence frequency ($m \ge 2$) and TF-IDF rarity to capture specialized terminology without common-word noise.
  5. Citation Co-occurrence (Implicit links): Pulls in papers frequently co-cited with already-retrieved literature to uncover latent relationships.
  6. Clustering: Groups the resulting candidate pool (~500+ papers) by cosine similarity and keeps only one representative paper per cluster to minimize redundancy.
  7. Dual-Path Idea Proposal:
  8. SciPIP-A (Direct): Summarizes the core problem, generates ~10 literature-informed candidate ideas, filters down to 5–6, and refines them into 3–4 detailed proposals with algorithms/equations.
  9. SciPIP-B (Dual-Path Independent): Concurrently runs the direct pipeline and an unconstrained brainstorming branch, merging both sets before filtering.
  10. SciPIP-C (Dual-Path Integrated): Incorporates keywords extracted from brainstormed directions into the literature retrieval step itself before merging and refining.

Datasets

  • Pre-indexed Database: 48,895 top-tier papers from ICLR, NeurIPS, ICML, ACL, NAACL, and EMNLP (2014–2024).
  • Evaluation Benchmark: 1,968 accepted papers from ACL 2024 (long, short, findings, and workshops), using their backgrounds as test inputs.
  • Novelty Verification: Live querying via the Semantic Scholar API to compare proposed ideas against external scientific literature.

Key Insights and Ideas

  • Retrospective Validation of Human-Level Ideation: By feeding in the background problem of accepted ACL 2024 papers, SciPIP frequently reconstructed the core idea of the actual published paper (similarity score 4/5), proving LLMs can match human research trajectories.
  • Low Similarity $\neq$ Low Quality: Ideas with lower similarity to published papers scored significantly higher in win-rate novelty evaluations (a 40.2% win rate for score-0 vs. 10.2% for score-4), while maintaining comparable feasibility. Non-matching ideas often represent valuable unexplored research paths.
  • The Paradox of Parametric Brainstorming: Unconstrained brainstorming (SciPIP-B/C) scored slightly lower on automated novelty benchmarks than pure literature-based prompting (SciPIP-A). The authors hypothesize that open brainstorming taps directly into the LLM’s pre-training biases and common patterns, whereas structured multi-hop retrieval forces the model to synthesize disparate concepts.

Key Results

  • Retrospective Idea Matching: Evaluated on ACL 2024 backgrounds, SciPIP variants generated 4 to 5 highly matching ideas (Score 4) per 100 backgrounds (and 91 Score-4 ideas across the full 1,968 set for SciPIP-C). In contrast, the baseline AI Scientist achieved 0 Score-4 ideas (peaking at Score 3).
  • Novelty Verification (vs. Semantic Scholar): Evaluated on a 0–10 scale by GPT-4o against real-world search results, SciPIP-A produced 92 ideas with a near-perfect novelty score of 9, compared to only 12 from AI Scientist. Across all variants, SciPIP generated over 270 high-scoring ideas ($>7$).
  • Literature Retrieval Recall: On predicting ground-truth core citations, SciPIP outperformed both SCIMON-like and ResearchAgent-like retrieval algorithms across all thresholds, achieving Recall@10 of 0.419 (vs. 0.381 and 0.377) and Recall@50 of 0.684 (vs. 0.616 and 0.622). Ablations confirmed that combining semantics, entities, co-occurrence, and clustering delivered the highest recall.

Limitations and Future Work

  • LLM-as-a-Judge Reliance: Both idea similarity and novelty scoring rely heavily on LLM evaluation (GPT-4o), which may carry implicit biases.
  • Domain Scope: Experiments were confined to the machine learning and natural language processing literature; cross-domain transferability (e.g., biology or physics) remains untested.
  • Mechanistic Understanding: The authors note an open question for future work: deeper exploration is needed to understand why ideas with lower retrospective similarity to existing papers consistently exhibit higher novelty without dropping in feasibility.

Summary updated

Abstract

The exponential growth of knowledge and the increasing complexity of interdisciplinary research pose significant challenges for researchers, including information overload and difficulties in exploring novel ideas. The advancements in large language models (LLMs), such as GPT-4, have shown great potential in enhancing idea proposals, but how to effectively utilize large models for reasonable idea proposal has not been thoroughly explored. This paper proposes a scientific paper idea proposer (SciPIP). Based on a user-provided research background, SciPIP retrieves helpful papers from a literature database while leveraging the capabilities of LLMs to generate more novel and feasible ideas. To this end, \textbf{1)} we construct a literature retrieval database, extracting lots of papers' multi-dimension information for fast access. Then, a literature retrieval method based on semantics, entity, and citation co-occurrences is proposed to search relevant literature from multiple aspects based on the user-provided background. \textbf{2)} After literature retrieval, we introduce dual-path idea proposal strategies, where one path infers solutions from the retrieved literature and the other path generates original ideas through model brainstorming. We then combine the two to achieve a good balance between feasibility and originality. Through extensive experiments on the natural language processing (NLP) field, we demonstrate that SciPIP can retrieve citations similar to those of existing top conference papers and generate many ideas consistent with them. Additionally, we evaluate the originality of other ideas generated by SciPIP using large language models, further validating the effectiveness of our proposed method
BibTeX
@misc{wang2024scipip,
  author = {Wenxiao Wang and Lihui Gu and Liye Zhang and Yunxiang Luo and Yi Dai and Chen Shen and Liang Xie and Binbin Lin and Xiaofei He and Jieping Ye},
  title = {{SciPIP: An LLM-based Scientific Paper Idea Proposer}},
  year = {2024},
  eprint = {2410.23166},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.23166},
}
Figure