Language-model systems that propose research ideas by borrowing a method from another field fail in a specific and repeated way. When a draft idea is rejected, because it duplicates an old paper or because the borrowed method cannot work in the new setting, the system rewrites the latest draft and tries again. It never asks which ingredient caused the rejection. Constraint solvers learned to avoid this behavior decades ago. Solvers that only undo their most recent choice (chronological backtracking) waste exponential effort when the real culprit is an early choice. Solvers that trace the failure to its cause, jump straight back to that choice, and record the bad combination so it is never tried again are dramatically faster. We propose to build that second kind of search for cross-domain ideation. A draft idea is treated as a set of explicit choices: the framing of the problem, the borrowed mechanism, the older work it builds on, and the evaluation. Every rejection must name a small set of choices that is verified to be responsible. The search jumps back to the most recent of those choices, and the culprit combination is stored as a permanent "nogood" that is reused across branches and across problems.
The opportunity is sharpest because of a puzzling negative result. In the PaperGym study, carefully targeted cross-domain retrieval produced proposals that won only 47% of novelty comparisons against proposals built from randomly chosen seeds (Read, Grep, and Synthesize). The study also found that models often "abandon" the borrowed paper: they cite it but never use its mechanism. Our claim is that this gap comes partly from the pipeline being feed-forward. Nothing in the system ever checks whether the borrowed mechanism actually carries weight in the proposal. A conflict analysis that tests this by swapping the ingredient out, and backjumps when the check fails, should turn retrieval from a random catalyst into a real dependency.
Core Claim
An ideation search that extracts verified conflict sets and keeps a persistent store of nogoods should do two things at a fixed budget of model calls. First, it should beat the PaperGym random-seed control, which targeted retrieval could not do. Second, it should reach accepted proposals with fewer generator and judge calls than chronological depth-first or value-based tree search. The new ingredient is not "use a critic," since every existing system has one. It is the verification of blame by counterfactual swaps, combined with reuse of the resulting constraint far from where it was learned.
There are two operational definitions. A conflict set is a subset of an idea's choices such that replacing any member with an alternative removes the failure and replacing non-members does not. A nogood is a generalized record of that subset, for example "a contrastive-alignment mechanism applied to a retrieval-ranking framing collides with a named prior paper." Scientific history enters the method through the novelty checker. It searches older literature as well as recent papers, so that "reinventing a 1990s idea," the pattern Schmidhuber's lineage note documents for adversarial networks, becomes a recorded, reusable conflict rather than a missed one.
Relation to Prior Work
The search machinery comes from classical work. Doyle's truth maintenance system recorded a justification for every belief and traced contradictions back to the assumptions responsible (Truth Maintenance Systems for Problem Solving). Prosser's conflict-directed backjumping keeps a conflict set for each variable and jumps to its deepest member (Domain Filtering can Degrade Intelligent Backtracking Search). Dechter showed that recording conflicts as new constraints gave a 5–10x speedup over backjumping alone on the Zebra puzzle (Learning While Searching in Constraint-Satisfaction-Problems). Kambhampati's minimal failure memos in planning reduced Rocket-ext-a backtracks from over 8,000 to 764 (Improving Graphplan's Search with EBL & DDB Techniques). Beame, Kautz and Sabharwal explain why such reuse matters: learned clauses carried across unrelated branches are what lift these solvers above tree-like search (Understanding the Power of Clause Learning). We reuse these techniques. The adaptation that is genuinely new is that our "constraint checker" is a noisy language-model judge. For that reason, conflict sets must be verified by experiment rather than read off a deduction.
Two classical results constrain the design. First, recording nogoods can cost more than it saves, as Dechter observed on easy problems and as the cost of matching Graphplan memos showed, so the overhead must be charged against the method's budget. Second, Prosser found that aggressive pruning can remove "bridge" values whose quick failure would have exposed a deep conflict. In our setting, the analogue is a semantic filter that throws away distant-looking seeds before they are tried. We therefore test pre-filtering explicitly rather than assuming it helps.
The ideation systems motivate the work, and each lacks one specific piece. Tree of Thoughts runs depth-first search over model "thoughts" but backtracks one level at a time and caches no reason for failure. IRIS uses Monte Carlo tree search with scalar rewards, so blame is spread over ancestor nodes rather than located in one. SCIMON feeds overlapping papers back as negative examples for the next attempt only, with no persistent memory and no decomposition of the idea. Chain of Ideas orders a single citation lineage in time and revises after review without locating a cause. ResearchTown simulates a research community by passing messages forward only. PaperGym supplies the cross-domain setting and the diagnosis, but it has no recovery step at all. None of these systems combines root-cause localization with cross-branch reuse, and that combination is what we isolate.
Research Program
Each candidate idea assigns values to four slots: problem framing, borrowed mechanism (a seed from the PaperGym library), precursor (an older paper the idea builds on), and evaluation protocol. Slots are filled in a fixed order, and each assignment is a decision level.
After every complete assignment, three checks run. The first is a novelty check, using the PaperGym-style judge with live Semantic Scholar search, extended back before 2017. The second is a feasibility check, asking whether the mechanism's assumptions hold for the framing. The third is a load-bearing check. We regenerate the proposal with the borrowed mechanism replaced by a random seed. If the judged quality does not drop, the mechanism was decorative and the idea fails. This is citation abandonment turned into a testable failure.
On failure, the judge proposes a candidate culprit subset. We then verify it with counterfactual swaps: each proposed member is replaced by an alternative while the other slots are kept fixed. If necessary, we also test non-members. Only slots whose replacement removes the failure stay in the conflict set. The search jumps to the deepest slot in the verified set. The set is generalized, with the seed abstracted to its mechanism type and the framing to its task family, and then written to a global nogood store shared across all problems. Before any expensive check, new assignments are matched against the store.
To keep overhead bounded, nogoods are forgotten when they go unused, the open problem Kambhampati identified. Every verification call counts against the same budget as generation.
Baselines
The naive reference point is single-shot synthesis: one proposal per problem from the same model and the same retrieved seeds, with no search. It tells us what any search buys at all.
The strongest existing method on this exact question is the PaperGym cross-domain retrieval and multi-seed synthesis pipeline (Read, Grep, and Synthesize), together with its random-seed control. The reported figures are a 47% pairwise novelty win rate against random seeds, 67% against same-domain retrieval, and 60% against no retrieval, with Claude 3.5 Sonnet as the judge. We rerun both arms ourselves with the same seed library, the same judge, and the same generator model, so comparisons are like for like. Beating the baseline means a win rate against the random-seed control whose bootstrap interval (resampled over problems) excludes 50%. Beating it also means a lower measured abandonment rate.
The search baselines are a depth-first Tree of Thoughts search over the same four slots with chronological backtracking, an IRIS-style Monte Carlo tree search with scalar review rewards, and a SCIMON-style loop that feeds back retrieved overlapping papers as negative examples. For none of these does Lacuna report a result on the PaperGym benchmark (no reported result found on Lacuna for ToT, IRIS or SCIMON on PaperGym), so we report all of them from our own reruns. Each receives an identical budget of generator plus judge calls, and each is tuned with the same number of configuration trials.
Three ablations isolate the key ingredients. The first is critique-and-rewrite: the judge names a culprit slot, and that slot is resampled with no verification and no memory. The second is verified backjumping with the nogood store cleared for every problem. The third is the full system with a semantic pre-filter that discards seeds below a similarity threshold, which tests the bridge concern.
Why not just…?
Why not just let the judge name the faulty component and rewrite it? This is the most serious threat, because it is cheap and may capture most of the benefit. The answer depends on how reliable the judge's blame is. Tree of Thoughts reports that self-evaluation by the model is imperfect, and an unverified culprit leads the search to jump to the wrong level. The critique-and-rewrite ablation tests this directly. If it matches the full system at equal budget, we will conclude that counterfactual verification is unnecessary and say so. We will also report how often the judge's proposed culprit survives verification.
Why not just give Monte Carlo tree search or depth-first search more budget? Larger budgets do help undirected search eventually. The classical argument is that chronological search re-explores doomed subtrees many times when the culprit lies early, which is the pattern behind the gap between 24 hours for chronological DPLL and 59 seconds for clause learning on pebbling formulas (Understanding the Power of Clause Learning). We plot quality against calls for every method. If the curves converge within realistic budgets, the advantage is smaller than claimed.
Why not just use more or more diverse seeds, since random seeds already work? PaperGym's parity result can be read this way: diversity is what matters. But Chain of Ideas found that an unordered pile of papers produced shallower ideas than an ordered chain, and PaperGym itself documents abandonment. Our load-bearing check makes the question measurable. If random-seed proposals with a larger seed count match ours on both win rate and abandonment, the "why gap" is not a dependency problem, and our hypothesis is wrong.
Proposed Experiments
The main study uses the 30 PaperGym problems and the 1,167-seed library, with five independent runs per method per problem at three budgets. Outcomes are four measures. The first is pairwise novelty win rate against each baseline, using the PaperGym judge, with a human-rated subsample to check judge agreement; Chain of Ideas reports 74.9% agreement between GPT-4o and human experts in a comparable setting. The second is calls to the first accepted proposal. The third is the abandonment rate under the swap test. The fourth is the fraction of novelty failures that resolve to pre-2017 precursors, which measures how much the historical extension contributes.
For generalization, we use the SCIMON gold test set of 194 instances with its temporal split, where human ideas beat SCIMON 85% of the time (SCIMON). The question is whether our method narrows that gap.
A mechanics sanity check uses Mini Crosswords, where constraints between words provide ground-truth conflicts. There, Tree of Thoughts solved 4 of 20 games with 60% word-level success. Verified backjumping should solve more games at equal calls, or else the search component is not doing what we claim.
The decisive checks are three. The full system must beat the random-seed control where retrieval alone tied it. It must beat critique-and-rewrite, which shows that verification matters. And it must beat the variant with nogoods cleared on later problems in a run, which shows that cross-problem reuse matters. Separately, the pre-filter ablation tells us whether Prosser's bridge effect appears in ideation. If the bridge effect holds, filtering should lower the rate of informative early failures and lengthen searches.
References
- Read, Grep, and Synthesize: Diagnosing Cross-Domain Seed Exposure for LLM Research Ideation
- Generative Adversarial Networks are special cases of Artificial Curiosity (1990) and also closely related to Predictability Minimization (1991)
- Truth Maintenance Systems for Problem Solving
- Domain Filtering can Degrade Intelligent Backtracking Search
- Learning While Searching in Constraint-Satisfaction-Problems
- Improving Graphplan's Search with EBL & DDB Techniques
- Understanding the Power of Clause Learning
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- IRIS: Interactive Research Ideation System for Accelerating Scientific Discovery
- SciMON: Scientific Inspiration Machines Optimized for Novelty
- Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents
- ResearchTown: Simulator of Human Research Community