Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas

Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan, Zhenzhong LanarXiv 2024 · DOI · arXiv

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in synthesizing existing knowledge, yet they often struggle with genuine scientific innovation. When tasked with generating research ideas, these models frequently encounter a "diversity wall," resulting in repetitive or incremental suggestions. This limitation stems from a lack of goal-oriented external knowledge acquisition; standard Retrieval-Augmented Generation (RAG) typically retrieves similar content rather than the "distant" knowledge required for creative breakthroughs.

Methodology: The Nova Pipeline

To address these challenges, we introduce Nova, a framework designed to proactively identify and retrieve external knowledge to fill reasoning gaps. The Nova pipeline consists of three distinct stages:

  1. Initial Seed Idea Generation: Utilizing a "seed paper" and 10 fundamental scientific discovery theories (e.g., Kuhn’s paradigm shifts), the system prompts the LLM to adopt the perspective of a philosopher of science to generate initial concepts.
  2. Iterative Planning and Search: This core stage involves multiple iterations where the LLM devises a search plan to identify specific fields or papers that could enhance novelty. It then retrieves these papers via the Semantic Scholar API and uses self-correction to refine the idea pool.
  3. Idea Completion: Promising seed ideas are expanded into comprehensive research proposals, including methods and experimental plans.
The Nova Pipeline architecture, illustrating the flow from initial seed generation through iterative refinement to final idea completion.

Experimental Evaluation

Nova was evaluated using a dataset of 170 high-quality papers from top-tier 2024 conferences (CVPR, ACL, ICLR). It was compared against state-of-the-art (SOTA) baselines, including AI-Researcher, AI-Scientist, and Research-Agent.

Key Results

The effectiveness of Nova's iterative approach is evident in both automated and human evaluations:

  • Novelty and Diversity: Nova produced 3.4 times more unique novel ideas than baselines without iterative planning. As shown in the performance data, the number of unique novel ideas increases significantly with each iteration step.
  • Tournament Performance: In a "Swiss Tournament" head-to-head ranking, Nova generated 2.5 times more top-rated ideas (619) compared to the next best method, AI-Scientist (249).
  • Human Validation: PhD-level experts confirmed that Nova contributed 37.5% of the "Top 4" ideas in comparative trials, while maintaining the lowest frequency of poor-quality ideas.
Comparative performance of Nova against SOTA baselines (left) and the impact of iteration steps on the generation of unique novel ideas (right).

Discussion and Conclusion

The success of Nova highlights that the strategy of search is more critical than the volume of search. By forcing the model to plan its information needs before retrieval, Nova avoids the echo chamber of its training data. While gains tend to plateau after three iterations, future work could involve formal reward functions or human-in-the-loop "co-pilot" configurations to further push the boundaries of AI-driven scientific discovery.

Summary updated

Abstract

Scientific innovation is pivotal for humanity, and harnessing large language models (LLMs) to generate research ideas could transform discovery. However, existing LLMs often produce simplistic and repetitive suggestions due to their limited ability in acquiring external knowledge for innovation. To address this problem, we introduce an enhanced planning and search methodology designed to boost the creative potential of LLM-based systems. Our approach involves an iterative process to purposely plan the retrieval of external knowledge, progressively enriching the idea generation with broader and deeper insights. Validation through automated and human assessments indicates that our framework substantially elevates the quality of generated ideas, particularly in novelty and diversity. The number of unique novel ideas produced by our framework is 3.4 times higher than without it. Moreover, our method outperforms the current state-of-the-art, generating at least 2.5 times more top-rated ideas based on 170 seed papers in a Swiss Tournament evaluation.
BibTeX
@misc{hu2024nova,
  author = {Xiang Hu and Hongyu Fu and Jinge Wang and Yifeng Wang and Zhikun Li and Renjun Xu and Yu Lu and Yaochu Jin and Lili Pan and Zhenzhong Lan},
  title = {{Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas}},
  year = {2024},
  eprint = {2410.14255},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.14255},
}
Figure