Large Language Models (LLMs) have emerged as remarkably creative engines for scientific brainstorming. When prompted to generate new research directions, they frequently outpace human experts in sheer originality and conceptual breadth. However, this creative explosion masks a fundamental bottleneck: a severe ideation-execution gap. While an AI-generated research proposal might sound brilliant in an abstract, it frequently collapses when subjected to real-world implementation, data constraints, or rigorous experimental validation.
Understanding why this gap exists—and how the field is moving to bridge it—requires looking at the fundamental trade-off between conceptual novelty and practical feasibility.
The Core Tension Between Originality and Utility
Why do AI models struggle to propose ideas that are both groundbreaking and doable? In both cognitive psychology and language modeling, there is an inherent negative correlation between how original an idea is and how practical it proves to be.
As shown by Claire E Stevenson et al. in Putting GPT-3's Creativity to the (Alternative Uses) Test, optimizing for extreme divergence naturally drifts away from practical utility. In the scientific domain, Sikun Guo et al. formalized this via IdeaBench in IdeaBench: Benchmarking Large Language Models for Research Idea Generation, demonstrating a consistent mathematical trade-off where increased novelty in LLM-generated ideas directly corresponds to decreased feasibility.
Human experts naturally filter their creative process through a lens of physical, financial, and logistical constraints. LLMs, by contrast, optimize for semantic plausibility. As Chenglei Si et al. observed in Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers, AI-generated ideas are often judged as significantly more novel than human ones, yet they consistently lag behind in practical feasibility because the models lack an inherent understanding of real-world constraints.
The Ideation Execution Collapse
The true extent of the problem only becomes clear when moving from the pitch to the pipeline. In The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas, Chenglei Si et al. tracked what happens when expert researchers attempt to fully execute LLM-generated proposals versus human-generated ones. While the AI ideas initially outshone human ideas at the abstract stage, their quality and feasibility experienced a massive decline during actual execution, ultimately reversing the rankings.
This discrepancy occurs because LLMs excel at generating ideas that appear technically sound but fail to yield actual performance gains or viable experimental designs when translated into code or tested on hardware, a phenomenon detailed by Chenglei Si et al. in Towards Execution-Grounded Automated AI Research. Similarly, benchmarks like AAAR-1.0 by Renze Lou et al. in AAAR-1.0: Assessing AI’s Potential to Assist Research and evaluations by Long Li et al. in Chain of Ideas: Revolutionizing Research in Idea Development with LLM Agents confirm that the experimental procedures proposed by AI agents are frequently unfeasible, vague, or logistically impossible compared to human-designed experiments.
The Grounding Paradox and the Pareto Frontier
If the problem is a lack of practical grounding, the obvious solution is to force the LLM to anchor its ideas in existing literature or data. However, this introduces a secondary paradox: grounding fixes feasibility but kills novelty.
For instance, Xiao Liu et al. demonstrated in Augmenting Research Ideation with Data: An Empirical Investigation in Social Science that explicitly providing models with dataset metadata significantly improves the empirical feasibility of their hypotheses, yet it simultaneously reduces the originality of the concepts. When models are heavily constrained by what data is readily available, they default to safe, predictable questions.
This tension between factual truthfulness and scientific novelty is a major hurdle. As Guangzhi Xiong et al. discovered with the TruthHypo benchmark in Towards Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Models, systems that prioritize high groundedness in existing literature avoid hallucinations but struggle to generate the bold, frontier-pushing insights necessary for true discovery. Furthermore, Wenxiao Wang et al. noted in SciPIP: An LLM-based Scientific Paper Idea Proposer that an LLM's brainstorming effectiveness is often inversely related to its ability to retrospectively match existing literature, highlighting how difficult it is to achieve both dimensions simultaneously.
Without explicit reasoning chains or provenance, as explored by Rosni Vasu et al. in HypER: Literature-grounded Hypothesis Generation and Distillation with Provenance, or rigorous metrics like those proposed by Sandeep Kumar et al. in Can Large Language Models Unlock Novel Scientific Research Ideas?, automated ideation remains unanchored.
Closing the Loop with Empirical Verification
To break this bottleneck, recent architectures are moving away from single-shot generation toward closed-loop execution and rigorous rejection protocols. Instead of treating ideation as a pure creative task, frameworks are embedding models within environments that force them to confront execution failures.
For example, the DOLPHIN framework by Bowen Zhou et al. in Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback introduces a traceback-guided debugger and an experimental feedback loop, allowing the AI to iteratively refine its proposals based on actual execution failures.
To prevent flawed ideas from entering the pipeline in the first place, protocols like NOVA-Test by David Scott Lewis et al. in NOVA-Test: Auditing LLM-Generated Research Hypotheses with Structural Analogy, Prior-Work, and Falsifiability Tests subject generated hypotheses to structural analogy, literature audits, and explicit falsification plans before execution.
Finally, systems like DeepScientist by Yixuan Weng et al. in DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively treat discovery as a goal-driven optimization problem, using a persistent memory of experimental findings to progressively navigate the full research cycle. By shifting the AI's objective from generating plausible text to surviving empirical verification, these frameworks aim to bridge the ideation-execution gap and unlock truly autonomous, viable scientific discovery.
Key Foundational Work
IdeaBench: Benchmarking Large Language Models for Research Idea Generation (2025) — contributed 4 concept elements to this direction.
Chain of Ideas: Revolutionizing Research in Idea Development with LLM Agents (2025) — contributed 2 concept elements to this direction.
SciPIP: An LLM-based Scientific Paper Idea Proposer (2025) — contributed 2 concept elements to this direction.