Scoring Cross-Domain Research Ideas Against microRNA Knockdown Data

Tools that use language models to generate research ideas are almost always graded by another language model. That makes one basic question hard to answer: does borrowing a mechanism from a distant field actually help, or does it only make a proposal sound new? The best diagnostic we have is PaperGym. Read, Grep, and Synthesize found that targeted cross-domain retrieval beats same-domain retrieval, but only ties a random-seed control (a 47% win rate). The same study found that models often "abandon" the borrowed idea: they cite it but never build it into the method. Both findings rest on judge scores. So we cannot tell whether targeted borrowing is really useless, or whether the judge simply cannot see the difference.

We propose to settle this with a scientific problem where ideas can be run as code and checked against real data. The problem is predicting how a microRNA (a short RNA that dials down many genes at once) represses its hundreds of targets. This spread of effects across many targets is called pleiotropy. Pleiotropy is a good testbed because simple one-target reasoning fails in documented ways, and because the field's biggest advances came from borrowing ideas from other disciplines. The ideation agent must write each proposal as a predictive model. That model is fitted and scored on held-out knockdown measurements. We then remove the borrowed component to test whether it actually carries weight.

Core Claim

Our claim is that PaperGym's random-seed tie is partly an artifact of judging text rather than results. Under an empirical score, targeted cross-domain borrowing should beat random seeds, but only when the proposal is forced to build the borrowed mechanism as a separate code module that can be removed. Without that requirement we expect to reproduce the tie, because borrowed ideas get abandoned.

The new ingredient is a quantity we call the binding rate. It is the fraction of proposals whose held-out accuracy drops when the borrowed module is swapped for a neutral default. This turns "citation abandonment" from something a reader notices into a number we can measure.

Why microRNA Pleiotropy Is the Right Test

The biology supplies failures that any correct model must explain.

The first is that single-target rescue does not work. MicroRNA-144 inhibits the metastasis of gastric cancer by targeting MET expression showed that miR-144 blocks cancer cell migration by repressing MET. Yet restoring MET only partly reverses the effect, because other targets such as ADAM12 and VCAN are repressed in parallel.

The second is that repression is nonlinear. Characterization of the microRNA pool found that repression rises steeply with the number of microRNA binding sites on a transcript and then plateaus above about 5 sites. It also found that the cell's pool of microRNAs buffers small changes, and that decoy transcripts (called competing endogenous RNAs) soak up active microRNAs.

The third is that broad, weak repression matters at the network level. Gene regulatory network stabilized by pervasive weak repressions imported May's stability theory from ecology. It argued that weak repression spread across 1–25% of genes stabilizes the whole gene network in a way that a few strong repressions cannot. That paper is our model of a successful hand-made borrowing across fields. We want to know whether an agent can find such borrowings on purpose.

TargetNet is another such borrowing. It brought residual convolutional networks from computer vision to the task of classifying binding sites, including sites with imperfect pairing. That paper also supplies our data: the miRAW site-level benchmark and 13 datasets measuring log fold change (LFC), the change in each target's expression after a microRNA is added to cells. TargetNet scores each binding site on its own. It does not model the pool of competing targets, which is the gap our prediction task exercises.

We also take a constraint from Virtual Cell Models Inflate Perturbation Effect Sizes, which found that learned cell models overstate how large perturbation effects are. For that reason we score how well predicted effect sizes are calibrated, not only how well targets are ranked.

Research Program

The prediction task. For each of the 13 LFC datasets, a model receives the added microRNA and the 3' untranslated region sequences of all measured transcripts. It must predict the LFC of every transcript. As input, it may use TargetNet's per-site scores and site positions.

How models are scored. Evaluation is leave-one-dataset-out, so each model is tested on a dataset it never saw during fitting. We report three numbers per held-out dataset:

  • Spearman correlation between predicted and observed LFC across all transcripts.
  • The slope of observed LFC regressed on predicted LFC, which checks calibration (a slope far below 1 means effects are overstated).
  • Whether the plateau at about 5 sites, as reported by Cui et al., appears when the fitted model is applied to reporter-like inputs.

The reporter data has only 23 genes, so the third check is a qualitative screen, not a headline number.

The ideation arms. We run a PaperGym-style agent that can read, grep, and run bash commands. Each arm draws seeds from a different source:

  • no retrieval;
  • same-domain microRNA literature;
  • random seeds from PaperGym's 1,167-seed library, extended with a few hundred seeds we extract from chemical kinetics, ecology, economics of shared resources, and control theory;
  • targeted cross-domain retrieval using PaperGym's domain paraphrasing, where the problem is restated in each field's vocabulary;
  • targeted retrieval with binding required.

Under binding, the proposal must implement the seed's mechanism as a named module with a declared neutral replacement. Examples include a competitive-sharing allocation rule or a saturating occupancy law.

Every arm gets the same budget: the same base model, the same number of proposals per problem, the same fitting compute, and the same number of debugging turns. Each proposal is fitted on the training datasets only.

What we record. For every proposal we record its held-out score, its binding status after ablation, and a PaperGym-style judge score for novelty and feasibility. The judge scores let us measure directly how well judge ratings track empirical results, which is the reward-hacking worry raised by IRIS.

Baselines

Do-nothing reference. This model predicts LFC as a monotone function of the canonical seed-site count, in the style of TargetScan. On miRAW, TargetScan's reported classification F1 is 0.4712, with recall of 0.3945, in TargetNet. Every arm must clear this floor.

Strongest existing site model. This is TargetNet's per-site confidence, summed per transcript and mapped to LFC by a monotone fit on the training datasets. TargetNet reports F1 of 0.7739 on miRAW, against 0.7289 for miRAW's own method. The paper describes the correlation of its scores with LFC as "strong" across all 13 datasets, but no numerical LFC correlation value was found on Lacuna for TargetNet. We will therefore recompute that baseline ourselves under our exact data split.

Strongest learned competitor. This is a gradient-boosted regressor and a small neural regressor trained on the same inputs: TargetNet site scores, site counts, site density, and 3' untranslated region length. Its hyperparameters are tuned with a search budget matched to the compute of one ideation arm. This is the model a skeptic would reach for first.

Expert mechanistic model. This is a hand-written titration model. The microRNA pool is shared among all expressed sites, so repression of each target is diluted as more sites compete, and each transcript saturates as its sites fill. It encodes exactly the observations of Cui et al., and it tests whether the agents add anything beyond known biology.

Key ablation. This compares targeted retrieval with and without the binding requirement. It isolates whether mechanism binding, rather than retrieval itself, drives any gain.

What counts as beating a baseline. An arm beats a baseline if its best-of-k held-out Spearman exceeds that baseline on a majority of the 13 held-out datasets, with a paired test across datasets, and if its calibration slope is no worse.

Why not just…?

Why not just fit a strong regressor and skip ideation? This is a real threat. For ranking transcripts inside a single dataset, a tuned gradient-boosted model may match anything the agents produce. Our bet is on transfer across datasets. Dilution by competing targets depends on how many sites are expressed in each cell setting, and a regressor trained without that structure should mis-scale effects on new datasets. The calibration slope under leave-one-dataset-out evaluation settles this. If the regressor wins on both correlation and slope, then the biology testbed has shown that borrowing adds nothing here, and we report that.

Why not just inject random seeds, since PaperGym found they tie? That tie was measured by a judge. Our design runs the random-seed arm under the same empirical score, and it reports the binding rate separately for each arm. If random seeds still tie targeted-plus-binding on held-out data, the "catalyst" reading of PaperGym is confirmed in a grounded setting, and our core claim is wrong.

Why not just prompt a strong model with microRNA papers? The same-domain arm is this alternative. PaperGym found that cross-domain retrieval beat same-domain retrieval 67% of the time on judged novelty, but novelty is not accuracy. A same-domain agent may simply rediscover the titration model, which is why that model is a baseline. The agents count as adding value only if they exceed it.

Decisive Checks

The main result is the held-out comparison of targeted-plus-binding against random seeds against the titration model.

The claim survives if two things hold. First, targeted-plus-binding must beat random seeds on held-out Spearman and calibration. Second, its binding rate must be clearly higher than that of targeted retrieval without binding.

A secondary check correlates judge scores with held-out scores across all proposals. A weak correlation would show that judge-based ideation benchmarks miss the property they claim to measure.

As a final sanity check, we ask whether the best agent-found models reproduce three documented findings without being trained on them: the plateau at about 5 sites, and the partial-rescue pattern for miR-144 predicted from repression of MET, ADAM12, and VCAN. Held-out transcripts are rescored with MET's term removed to check the second.