Automated research-ideation systems are large language model agents that read papers and propose new research. They increasingly claim to find ideas by importing a method from one field into another. We cannot yet tell whether they do this well, because every current evaluation asks another language model, or a small panel, whether the proposal "seems novel." The sharpest diagnostic so far shows why this is a problem. Read, Grep, and Synthesize found that retrieving methods from other fields beats retrieving from the same field (67% pairwise wins). Yet targeted cross-domain retrieval only ties with feeding the model randomly chosen papers (47%). Random-seed proposals often cite the imported paper without using its mechanism. A judge scoring novelty cannot separate a real transfer from a decorative citation.
We propose to evaluate cross-domain ideation against history instead of against a judge's taste. Hidden Markov model (HMM) speech synthesis is unusually well suited to this. It is the approach that generates speech by modeling acoustic parameters as probability distributions over states, and its history is a chain of dated, well-documented method imports whose outcomes were measured in listening tests. We freeze the literature just before each import. We then ask ideation systems to solve the problem as it was stated at the time, and score whether they recover the mechanism that actually worked. The measurement is mechanism recovery against a known answer. If targeted retrieval has real value that novelty judges miss, it should clearly beat random seeds here.
Core Claim
The 47% tie between targeted and random cross-domain retrieval is at least partly a measurement artifact. A benchmark built from time-sliced historical migrations in speech synthesis, scored on whether the imported mechanism is structurally present, will separate targeted retrieval from random exposure. We also expect it to reveal which ideation architectures actually integrate mechanisms rather than cite them. The contribution is the benchmark and the diagnostic. We are not proposing a new ideation agent.
Why HMM Synthesis Is the Right Testbed
Statistical parametric synthesis has several migrations that a benchmark can replay, each with a clear before-state, an identifiable source, and a measured outcome. The first is the core move itself: HMMs came from speech recognition, and synthesis was recast from storing and splicing recorded clips to generating parameter trajectories. Zen and Maia's system in An HMM-Based Brazilian Portuguese Speech Synthesizer shows the payoff. Trained on 18 minutes 48 seconds of speech, it reached a mean opinion score (MOS) of 3.2, against 1.7 for MBROLA.
The second migration fixed the flat, stepwise output of frame-independent HMMs. Velocity and acceleration features were tied to the static trajectory and solved jointly, as in An introduction of trajectory model into HMM-based speech synthesis. That work reports a 69.1% listener preference against 42.5% for Baum-Welch training.
Three more migrations complete the set:
- Speaker adaptation. Linear-regression and MAP adaptation, originally from recognition, were used to build new voices from little data. HMM adaptation and voice conversion for the synthesis of child speech found listeners preferred this approach to voice conversion.
- Hybrid synthesis. Statistically predicted trajectories were used as soft targets for unit selection in CMU Blizzard 2007. It reports MOS 3.38 against 2.98 for plain unit selection.
- Exports out of speech. Component-wise model substitution was applied to clinical voice repair in Using HMM-based speech synthesis to reconstruct the voice of individuals with degenerative speech disorders, and the whole pipeline was transferred to animal song in Bird Song Synthesis Based on Hidden Markov Models.
These papers play two roles here. They supply the ground truth, meaning what the right import was. They also supply outcome evidence that the import worked, which a novelty judge never has.
Research Program
We will build a benchmark of historical migration events. Each event has four parts: a cutoff date, a problem statement rewritten in the vocabulary available before the cutoff, a source pool of pre-cutoff papers from both the target and candidate source fields, and a mechanism rubric. The rubric lists the two to four structural components that define the historical solution. For the trajectory model, those components are an explicit constraint linking static and dynamic features, and a global solution for the whole utterance rather than frame-by-frame decoding.
The research file documents six events. Six is too few for statistical power, so we will curate further events from the speech literature: recognition, synthesis, voice conversion, and clinical speech. We aim for at least 40, each written by two speech researchers who agree on the rubric before any system is run. Where a later migration generalized an earlier one, we treat it as a separate event. Feedback we cannot obtain for an event disqualifies it.
Contamination is the central threat, because language models have read this history. We handle it in three ways:
- Report only within-model differences. The primary quantity is the gain from each retrieval condition over the same model with no retrieval. Memorization inflates all conditions of a given model roughly equally, so it largely cancels in the difference.
- Rename the terms. A vocabulary-obfuscation arm renames domain terms throughout the problem statement and source pool, for example calling "delta features" "first-difference channels" and "decision-tree clustering" "question-based parameter sharing." This removes lexical cues to the memorized answer.
- Probe the base model. A recall probe asks each model directly whether it can name the historical solution. We stratify all results by probe success.
Scoring has two parts. The first is mechanism recovery: the fraction of rubric components present in the method section. Two blind speech experts score it, and we report agreement with an automated judge. Chain of Ideas reports 74.9% judge-human agreement on its own task, which gives a reference level. The second is integration rate: the fraction of cited seed papers whose mechanism actually appears in the method. This operationalizes the citation abandonment that PaperGym described only qualitatively.
Baselines
The naive reference point is the same model given the problem statement and no retrieved literature. On historical events this also measures what the model has memorized. Every other condition is reported as a gain over it.
The random-seed control follows PaperGym's design. The model receives the same number of seeds, drawn uniformly from the pre-cutoff pool of all fields. This is the condition that tied targeted retrieval under novelty judging. The key comparison of the whole proposal is whether it still ties under mechanism recovery.
The strongest existing method for this exact question is PaperGym's targeted cross-domain retrieval with query paraphrasing, from Read, Grep, and Synthesize. It is the only system built specifically to import mechanisms across fields, and speech is one of its seven domains. Its reported results are a 67% win rate against same-domain retrieval, 60% against no retrieval, and 47% against random seeds, together with tool-augmented extraction specificity of 4.76/5.
We also run two structured-history agents:
- Chain of Ideas (Chain of Ideas) orders citation chains chronologically. It reports a 65 Elo margin over the second-best baseline.
- Tree-of-Ideas (Tree-of-Ideas) reasons across branching citation lineages. It reports 6.27/10 against 5.36 for ResearchAgent, but only on deep-learning topics.
No reported result was found on Lacuna for any of these systems on historical or speech-synthesis migrations.
Two further reference points bound the comparison from below and above. The embedding-based Swanson ABC association method of Automatic Detection of Cross-Disciplinary Knowledge Associations finds a missing link between two literatures through a shared middle concept. It reports 95.68% accuracy on novel associations in Learning Analytics. We use it to test whether term-level linking alone identifies the right source field. The oracle condition hands the model the actual historical source paper, giving an upper bound on recovery given perfect retrieval.
The fairness rules are as follows:
- All language-model conditions use the same base model, the same seed count, the same pre-cutoff pool, and the same token budget.
- Each condition gets the same number of prompt-tuning trials, run on a held-out set of five events.
- Five samples are drawn per event per condition.
Targeted retrieval "beats" random seeds if its mechanism-recovery gain over no retrieval is higher, with a paired bootstrap interval across events that excludes zero, in both the plain and the renamed-vocabulary arms. The obfuscation arm is the ablation that isolates the proposal's key ingredient, historical ground truth free of lexical memorization.
Why not just…?
Why not just keep judging novelty with a language model, as PaperGym and Tree-of-Ideas do? That protocol produced the 47% tie. The same paper observed that random-seed proposals abandon citations, a failure a novelty score does not penalize. We will run the PaperGym-style pairwise novelty judge alongside mechanism recovery on the same outputs. If the two metrics rank conditions identically, our benchmark adds nothing, and we will report that.
Why not just use ResearchBench's masked-paper reconstruction? ResearchTown already scores reconstruction against real papers, reporting 0.68 semantic similarity for paper writing, and we use the ResearchBench format as a reference. Its tasks, however, are 2024 papers scored by overall text similarity. A proposal can resemble the target paper's prose while missing its mechanism. It also does not isolate cross-field imports. Our rubric scores structural components, and our events are selected for being imports.
Why not just run Swanson-style term linking, which is far cheaper? This is a real competitor for the source-identification half of the task, and we include it as a baseline. It can say "recognition adaptation relates to voice building." It cannot say "tie the adapted Gaussian means through a shared regression tree." If term linking finds the correct source field as often as agents do, then agents only add value in the integration step, and the integration-rate metric will show whether they add any there.
Why not just argue contamination makes history useless? It may, for well-known events. If no-retrieval models recover most rubric components even under renamed vocabulary, the benchmark is saturated for those events, and we will say so. That result itself warns against historical evaluation. The probe stratification tells us which events remain informative.
Proposed Experiments
The decisive experiment is the targeted-versus-random comparison on mechanism recovery, run under renamed vocabulary on events the recall probe marks as unknown to the model. If targeted retrieval shows a significant gain there, the novelty-judge tie was a measurement artifact. If it still ties, the problem is in how agents reason about why a mechanism transfers, not in how they are evaluated.
A second check correlates each agent's integration rate with its mechanism recovery. The purpose is to test whether citation abandonment is what separates systems.
A small execution arm then ties the benchmark back to speech. We take the three highest-scoring non-historical proposals for the open problem of oversmoothing under scarce data. These are the problems that remain after the trajectory model, whose 69.1% preference result was obtained on 1,096 training sentences. We train them on a one-hour CMU ARCTIC subset and run a paired preference test against a trajectory-HMM baseline on held-out sentences. This asks whether benchmark scores predict proposals that actually work when built.
References
- Read, Grep, and Synthesize: Diagnosing Cross-Domain Seed Exposure for LLM Research Ideation
- An HMM-Based Brazilian Portuguese Speech Synthesizer and Its Characteristics
- An introduction of trajectory model into HMM-based speech synthesis
- HMM adaptation and voice conversion for the synthesis of child speech: a comparison
- CMU Blizzard 2007: a hybrid acoustic unit selection system from statistically predicted parameters
- Using HMM-based speech synthesis to reconstruct the voice of individuals with degenerative speech disorders
- Bird Song Synthesis Based on Hidden Markov Models
- Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents
- Tree-of-Ideas: Automated Research Ideation via Cross-Trajectory Reasoning over Scholarly Evolution
- Automatic Detection of Cross-Disciplinary Knowledge Associations
- ResearchTown: Simulator of Human Research Community