Grounding the LLM Judge: Resource-Aware Evaluation of Genetic Circuits

When a language model designs a genetic circuit — a set of DNA parts assembled in a living cell to perform a task like toggling between states or oscillating — nobody currently checks whether the circuit will actually work in a cell. The best existing system, GenCircuit-RL, trains language models to generate circuits in a standard format and verifies them through five programmatic levels: does the code run, is the format valid, are the parts assembled correctly, are they annotated with the right biological terms, and does the topology implement the intended logic. Its own authors state that this verification stops at structure and topology and does not simulate quantitative behavior such as oscillation periods or bistability thresholds. Meanwhile, two mechanistic modeling papers prove that this omission is not benign: a resource-dependent protein synthesis model shows that a topologically perfect repressilator stops oscillating once background load on shared ribosomes and RNA polymerases is accounted for, and a coarse-grained bacterial cell model shows that growth-rate feedback alone can create bistability in circuits that have no explicit cooperative interactions. A circuit that passes every structural check can still die in the host.

The second ingredient is a warning about evaluation itself. The LLM-as-a-judge literature — using large language models to grade open-ended outputs — has documented that judges reward surface style over substance. On the Kahneman4Review benchmark, an LLM judge rating academic peer reviews collapsed nine intended rubric dimensions into a single stylistic factor, and 74% of the apparent score advantage of AI-written reviews over human reviews was explained purely by text length and formatting. If we simply asked an LLM to grade genetic circuit designs from their descriptions, it would plausibly reward articulate but physically non-viable designs. The proposal here is to build the opposite kind of judge: an LLM that is not trusted to assess biological feasibility from text, but instead orchestrates mechanistic simulators, reads their numerical traces, and issues a graded verdict that must cite simulator evidence.

Core Claim

The claim is that LLM-as-a-judge and mechanistic biological modeling are complementary, and that the right architecture is a simulator-grounded judge: the LLM handles the parts that formal tools cannot — parsing fuzzy natural-language specifications, supplying parameter priors for parts without measured kinetics, choosing simulation conditions, and translating numerical traces back into graded, actionable design feedback — while all claims about physical viability come exclusively from the mechanistic simulators it invokes, never from the LLM's own biological intuitions. On top of this, the judge should be distributional: rather than emitting a single pass/fail or scalar score, it predicts a full distribution over verdict levels, aligned to the population-level distribution of behaviors that stochastic biological circuits actually produce. Distributional LLM-as-a-Judge showed that training judges to predict the distribution of human ratings via KL divergence cuts judge–annotator divergence from 0.60 to 0.23 on SNLI; here the reference distribution is not human opinion but the output distribution of an ensemble of stochastic simulations, where the "disagreement" being modeled is molecular noise rather than rater subjectivity.

Formally, for a candidate circuit $C$ and specification $S$, the judge outputs a distribution $p_\theta(\cdot \mid C, S)$ over an ordered verdict scale (e.g., five levels from "fails under any resource load" to "robust across the target host regime"). The reference distribution $q(\cdot \mid C, S)$ is constructed from an ensemble of stochastic simulations of a resource-aware model of $C$ under sampled host conditions, and the judge is trained to minimize $D_{\mathrm{KL}}(q \,\|\, p_\theta)$. The KL form comes directly from the distributional-judge result cited above, adapted so the target is physical rather than human.

Research Program

The system to be built — call it a resource-aware circuit judge — has five components, each reusing existing technique where it exists.

First, the structural front-end reuses the programmatic hierarchy of GenCircuit-RL unchanged: code execution, SBOL schema validity, part assembly, and ontology checks remain deterministic compiler checks with zero hallucination risk. Nothing about the proposal replaces these; they are cheap, exact, and already shown to matter (hierarchical verification improved functional task success by 14–16 percentage points over binary rewards on the SynBio-Reason benchmark).

Second, a model-compilation step, which is the LLM's first substantive job: translating a validated SBOL circuit plus its natural-language specification into a system of resource-aware differential equations. The target formalisms already exist. The ribosome-flow-model machinery of the resource-dependent protein synthesis model provides mass-conserving transcription–translation dynamics, and the coarse-grained cell model adds the host side: ribosomal, metabolic, and housekeeping proteome sectors coupled through ppGpp signaling, reproducing empirical bacterial growth laws. Default parameter values are calibrated from BioNumbers-style physiological constants — the resource model already replicates average RNA-polymerase occupation of roughly 0.77 per gene and ribosome density of roughly 3.4 per mRNA in typical E. coli. For parts with unknown kinetics, the LLM proposes parameter priors with explicit uncertainty, following the spirit of the Approximate Bayesian Computation strategy in Bayesian and Algebraic Strategies to Design in Synthetic Biology, which exists precisely because kinetic parameters are unknown for most parts and brute-force simulation over wide parameter boxes is computationally prohibitive.

Third, a stochastic ensemble simulator runs the compiled model across sampled host conditions (nutrient quality, background load, growth rate) with intrinsic noise, producing for each candidate circuit a population-level output distribution — not a single trajectory. This directly addresses what Tuning the dials of Synthetic Biology identifies as core biological complexity: parts are context-dependent and non-orthogonal, and cell populations exhibit wide, often multimodal, expression distributions.

Fourth, the distributional verdict: the LLM reads the simulated ensemble summary and predicts $p_\theta$, trained against the simulator-derived $q$ via the KL objective. When the LLM's parameter priors were used (because kinetics were unknown), the prior uncertainty propagates into a wider $q$, so the judge is penalized for false confidence — the judge learns to say "this design's viability is unresolvable without measuring two rate constants."

Fifth, an evidence-cited critique: every verdict statement must reference a specific simulation artifact (a bifurcation threshold, a dilution-induced bistability window of the kind the coarse-grained model predicts, an oscillation-death boundary of the kind the resource model proves exists). Verdict statements without a simulation citation are dropped. This is the structural defense against the "analytical form over epistemic function" failure documented in Kahneman4Review.

Baselines

The naive reference point is an ungrounded LLM judge: a frontier LLM given the same circuit descriptions and specifications in natural language, with no simulator access, asked to grade on a rubric. This is the "do nothing" baseline because it is what someone would deploy tomorrow with no additional infrastructure; Kahneman4Review predicts it will be dominated by style, but the size of that failure on biological designs is itself an open measurement. No reported result exists on Lacuna for an ungrounded LLM judging genetic circuit viability.

The strongest existing competitor is the programmatic five-level hierarchical verification of GenCircuit-RL (GenCircuit-RL), which reports a 14–16 percentage-point improvement in task success rates on SynBio-Reason functional reasoning tasks over models trained with binary rewards. It is the strongest competitor because it is the current state of the art for language-model-based circuit evaluation and because its verification is deterministic and cheap. It will be run exactly as published on the same circuit set, with identical compute and no additional tuning, extended only by its own authors' acknowledged limitation: we add the dynamic checks it explicitly does not perform. Beating it means distinguishing, among circuits that pass its Level 5 function check, those that survive resource loading from those that do not — with the survival label determined by ground-truth simulation, and showing the judge's ranking correlates with that label where Level 5 cannot (by construction, Level 5 is constant on that subset, so any discrimination above chance is a win).

The ablation that isolates the key ingredient is a pure pipeline with no LLM in the loop: a fixed script that compiles SBOL to the resource-aware ODE model with a hard-coded default parameter table, runs the same ensemble, and scores by a fixed distance to the target distribution. Same circuits, same simulator budget, same ensemble sizes. This ablation separates the LLM's contribution (spec parsing, parameter priors for unmeasured parts, graded critique) from the simulator's contribution. If the fixed-parameter pipeline matches the full judge on circuits where all kinetics are known, that is the expected result and sharpens the claim that the LLM matters exactly where parameters and specifications are fuzzy; if it matches even on the fuzzy subset, the LLM component is not earning its cost and the proposal should be cut back.

A secondary ablation replaces the distributional verdict with a single-point score, keeping everything else fixed, to test whether distributional training transfers from human-rating alignment (its KL 0.23 vs 0.60 result on SNLI) to physical-distribution alignment.

Why not just…?

Why not just run ODE simulators and skip the LLM entirely? This is the strongest objection, and the fixed-parameter ablation above is designed to answer it honestly. The mechanistic side is genuinely powerful: the resource-dependent model proves oscillation death under load with nothing but analysis, and the systematic promoter–RBS design methodology optimizes transfer functions with a plain genetic algorithm plus math. But a simulator-only pipeline needs two things it cannot supply itself: kinetic parameters for unmeasured parts — Bayesian and Algebraic Strategies exists precisely because these are missing for most parts — and an interpretation of semi-quantitative, free-text specifications and assay notes, which AI Challenges in Synthetic Biology Engineering identifies as the core bottleneck of the design-build-test cycle. The decisive experiment is the parameter-scarcity split: on circuits where all parameters are known, the LLM should add nothing (and we will report that); on circuits where 30–70% of parameters are unmeasured, the judge's calibrated priors should recover the correct viability label substantially more often than the fixed-parameter pipeline or uniform-prior sampling at matched simulation budget. If they do not, the objection stands.

Why not just extend GenCircuit-RL's verification with a Level 6 simulator check? A reviewer could fairly say: keep the purely programmatic philosophy, add a deterministic ODE integration step, and never let an LLM judge anything. This is a real threat, and the honest answer is that for the subset of circuits with known parts and crisp specifications, that hybrid is probably sufficient — our claim is explicitly not that an LLM is needed there. The differences emerge at the edges: a rule-based Level 6 cannot handle un-parameterized parts, cannot parse a specification like "should oscillate robustly in slow-growth media but stay off in rich media," and cannot produce graded, evidence-cited feedback that a designer or a downstream RL loop can act on — the silent-bias work and the bias-quantification work cut against trusting LLMs with unfettered judgment, which is exactly why ours is fenced by simulators. Experiment three below settles it: if the rule-plus-simulator Level 6 matches the grounded judge as a training reward, we adopt it.

Why not just use human experts? Human panels remain the gold standard for laboratory feasibility — CRISPR-GPT leaned on expert curation before wet-lab execution, and its wet-lab results (multi-gene knockout at ~80% efficiency with Cas12a, activation up to 90.2%) validate expert-in-the-loop practice. But the SynBio-Reason corpus alone contains roughly 4,753 circuits across six circuit types, and automated design loops generate far more; expert review does not scale to that volume, and Distributional LLM-as-a-Judge documents that human raters disagree with each other enough that single-point labels are the wrong target anyway. The compromise is built in: a calibration subset of a few hundred verdicts is rated by a small expert panel, and the judge's distributional alignment to that panel is reported alongside its alignment to simulation.

Why not just fine-tune the LLM judge on labels and trust it? Kahneman4Review makes this untenable: an LLM judge collapsed nine rubric dimensions into one stylistic factor, and its one encouraging result — correctly distinguishing genuine fault-finding from fluent vacuity in 33 of 35 controlled probe pairs — came only when the evaluation was anchored to checkable content. That result is our template: the judge here is anchored to simulator output by construction, and the style-probe experiment below tests the same failure mode in the biological domain.

Proposed Experiments

The first experiment measures the style bias of ungrounded judging on biological designs. We construct paired probes: for each pair, one design is described in fluent, jargon-rich language ("orthogonal modular cascade with minimal retroactivity") but is physically non-viable under resource loading, while the other is described tersely but survives the simulated ensemble. Pairs are built by taking SynBio-Reason circuits and adding background load until the resource model's oscillation-death or the coarse-grained model's burden-induced bistability kicks in. The ungrounded baseline judge should fail these pairs at above-chance rates in the direction of the fluent description; the grounded judge should track the simulator. The decisive statistic mirrors the function-probe design of Kahneman4Review (33 of 35 pairs, dz = +2.52), and the same probe set doubles as an adversarial test suite against known judge biases like position and verbosity effects.

The second experiment measures distributional alignment on a dynamic extension of SynBio-Reason. The existing benchmark's 4,753 circuits cover six circuit types including oscillators and toggle switches; we extend each functional task with a stochastic target specification (a target population distribution, bistability window, or oscillation-period band under sampled host conditions) and report the judge's KL divergence to the simulator-derived reference, against the single-point ablation. Whether the SNLI-scale improvement (0.60 to 0.23) transfers to physically-derived distributions is the open question; there is no reported result on Lacuna for distributional judging outside NLP benchmarks.

The third experiment closes the loop: the judge as a training signal. GenCircuit-RL trains with GRPO against its five-level programmatic reward; we retrain with the grounded distributional verdict appended as a sixth reward level and test whether the resulting generator produces circuits that, when simulated under resource load, survive more often than those trained with the original reward — the original paper's reported gain over binary rewards is 14–16 percentage points on functional tasks, but it has no reported result for behavior under host burden, because it never simulated any. The rule-plus-simulator Level 6 baseline is run under the same training budget, as is the ungrounded-LLM-reward variant, which the style-bias literature predicts will collapse into rewarding articulate designs.

The final check is expert calibration: a few hundred judged verdicts on the fuzzy-specification subset, rated blind by a small panel of synthetic biologists, reporting the judge's distributional alignment to the human panel and its calibration on the "unresolvable without more measurements" verdicts. The proposal succeeds if the grounded judge beats the strongest programmatic baseline at separating viable from non-viable circuits under resource load while the ungrounded judge demonstrably cannot; it fails, and should be reported as failing, if the fixed-parameter simulator pipeline matches it everywhere — in which case the durable contribution is the dynamic evaluation layer itself, and the LLM's honest role is narrower than claimed.