A Local, Self-Consistent Judge for Bioinformatics Agent Runs

Language-model agents can now run a full sequencing analysis from a plain-English request. Examples include aligning reads, calling variants, running differential expression, and clustering single cells. Each of these systems also depends on a second model that decides whether the run was any good. That second model is the "judge" or "evaluator." It is usually a large proprietary model asked to read the agent's code and logs and output a score. Its verdict has two weaknesses. Asked twice about the same run, it often gives different answers. And because it runs in the cloud, it cannot be used on the private genomic data that labs most need to analyze.

We propose to train a small, open-weight judge that runs on a lab's own machine and grades bioinformatics agent runs consistently. The student is taught to copy the average opinion of a large teacher judge rather than any single sample of it. This is knowledge distillation, meaning training a small "student" model to imitate a large "teacher." Training also includes pairs of runs that differ only in harmless ways, such as renamed files or reordered flags, and these must receive the same score. It includes runs with deliberately injected scientific errors, such as a wrong reference genome or swapped sample labels, and these must score lower. The claim is that consistency can be built into the student during training rather than bought at test time by repeated sampling. We expect the student to score runs more consistently than its teacher, to catch hallucinated analyses the teacher misses, and to run locally.

Why this, and why now

Three findings in the literature motivate the gap.

BioAgent Bench runs agents on ten real multi-step bioinformatics workflows under perturbations such as prompt bloat, corrupted inputs and decoy files. Its main findings are:

  • Open-weight agents lag proprietary ones (82.5% completion for the best open model versus 100% for Claude Opus 4.5).
  • Agents are highly variable across repeated trials (mean Jaccard overlap of results 0.43).
  • Agents often proceed to "analyze" corrupted data as if it were valid.

Yet the benchmark grades traces with an uncalibrated GPT-5.1 judge whose own variability is never measured.

Work on judge consistency explains why that matters. Evaluating the Consistency of LLM Evaluators separates two failures. Self-consistency is the same verdict across repeated runs. Inter-scale consistency is the same logic whether the score is binary, Likert or numeric. It finds that general capability does not predict either, and that a 7B Mistral judge was more consistent than GPT-3.5. Reference-Free Rating of LLM Responses via Latent Information reports that even Qwen2.5 70B agrees with its own modal score only 70–80% of the time. It shows that reading a probability-weighted score from the model's output logits, or training a probe on its hidden states, is more stable than parsing a generated number.

Finally, accessibility systems that need judges either rely on cloud models or avoid hard tasks. CellAgent uses a GPT-4o Evaluator to critique single-cell analyses. BioAgents runs locally on Phi-3 with ontology-grounded retrieval, but it fails on multi-step glue code and degrades after three rounds of self-reflection.

Two papers supply our method. FENICE breaks summaries into atomic claims and checks each with natural-language inference. It distilled GPT-3.5's claim extractor into a 700M T5 model at a modest cost (70.0% versus 72.7% balanced accuracy on AggreFact). That suggests decomposition survives compression. Certifying the Judge tests Lean 4 proof judges against four falsifiable properties:

  • Identity: the same input gets the same score.
  • Bug monotonicity: injecting a bug lowers the score.
  • Specification monotonicity: dropping a requirement lowers the score.
  • Invariance under renaming.

These properties are aggregated into a variance-aware trust index that reaches 0.833 Spearman correlation with human ratings. We reuse those properties, but as training signal for a student, not only as a certification test. We also translate them from formal proofs to pipeline glue code.

One paper constrains the design. The latent-rating study observes that fine-tuned judges such as Prometheus can have distorted output probabilities, which hurts logit-based scoring. Our student is fine-tuned, so calibration is a primary outcome we measure, not an assumption. Soft-target training is chosen specifically because it fits probabilities directly rather than hard labels.

Core Claim

A compact open judge, distilled from a repeatedly sampled frontier teacher, can beat its teacher's single-sample self-consistency on bioinformatics agent traces while staying close to the teacher's averaged accuracy. Two ingredients make this possible:

  • Soft targets: the empirical distribution over the teacher's K sampled verdicts.
  • Metamorphic training pairs: an invariance loss on harmless rewrites and a ranking loss on injected scientific errors.

The result is a judge that, unlike the teacher, behaves monotonically under the kinds of errors that matter biologically.

Research Program

Trace corpus. We run several agents repeatedly on every BioAgent Bench task and its perturbed variants: GLM-4.7, at least one smaller open model, and one proprietary model, each under one harness. The poor trial-to-trial overlap the benchmark reports is useful here. It yields many naturally different successful and failed runs per task.

Metamorphic variants. From each trace we generate scripted variants of three kinds:

  • Harmless rewrites that must not change the verdict: renaming intermediate files, reordering command-line flags, swapping equivalent tool invocations, and reformatting logs.
  • Injected errors that must lower it. These mirror the hallucination types catalogued in the literature: a mismatched reference genome build, sample or condition labels swapped in a DESeq2 design, a dropped normalization or filtering step, a fabricated tool flag, and a final report whose claimed numbers disagree with the output files.
  • Hallucinated analysis: the agent producing confident results on the benchmark's corrupted inputs, which the judge must penalize.

Teacher labeling. A frontier teacher judges every trace and variant K=8 times on a 1–10 numeric scale. The numeric scale was the more stable format in Lee et al. The teacher returns a total score and per-step atomic verdicts, following the claim-decomposition idea from FENICE. For example, one verdict might be "reads were aligned to GRCh38: supported by log line X." The target for the student is the empirical distribution over the K verdicts.

Student training. The student is an open 7–8B model; Mistral-Instruct-class is a natural choice given its reported consistency. We also train a Phi-3-mini-scale variant to match BioAgents' local footprint. The loss has three parts: cross-entropy to the teacher's soft score distribution, an invariance penalty on the score distributions of rewrite pairs, and a margin ranking loss on clean versus injected-error pairs. The ranking loss can make the student stricter than the teacher wherever the teacher fails to notice an injected bug. At inference we read the probability-weighted expected score from the student's logits, never a sampled string.

Human panel. Bioinformaticians label roughly 150 traces and their variants on a 1–10 scale. The scale matches the teacher's so that Spearman correlations are comparable. The size is comparable to the 75-item calibration panel in Certifying the Judge. These labels are used only for evaluation and for prediction-powered confidence intervals. That method, from Efficient Inference for Noisy LLM-as-a-Judge Evaluation, combines many noisy judge labels with few gold labels to produce valid intervals. It lets us state benchmark completion rates with honest error bars when any judge is used.

Baselines

Naive reference points. The first is the frozen student base model prompted with the same rubric and greedy decoding. This is what a lab would get today by downloading an open model; it isolates the effect of all training. The second, which needs no language model at all, is deterministic artifact checks against the benchmark's ground-truth outputs. These work only where ground truth exists.

Strongest existing method. The teacher itself, sampled K=8 times and averaged, is our accuracy ceiling. It is the repeated-sampling remedy in Evaluating Evaluation Metrics. We also report the teacher at a single sample at temperature zero, which is how BioAgent Bench uses GPT-5.1.

Strongest open, non-distilled competitor. We take the latent-information approach applied to an untuned 70B open model: a probability-weighted score plus a linear probe on hidden states. That paper reports about 0.6 Pearson correlation with human judgments for probability-weighted scores on general chat benchmarks. For judge consistency or human agreement on bioinformatics agent traces, no reported result found on Lacuna. For the only domain-matched setting, BioAgent Bench's GPT-5.1 judge, no reported consistency figure found on Lacuna. For reference outside the domain, the trust index in Certifying the Judge reaches 0.833 Spearman with human ratings on Lean 4 code.

Fairness rules. All judges see identical traces, rubrics and score scale. Any probe gets the same training traces as the student. Hyperparameter search uses the same budget of trials on a held-out validation split, with tasks split so no BioAgent Bench task appears in both training and test.

What counts as beating a baseline. The student beats a baseline if it does all three of the following. It has higher self-consistency (Krippendorff's alpha across K=5 runs) and higher inter-scale consistency than single-sample teacher and frozen base. It achieves a higher pass rate on held-out metamorphic pairs. And its Spearman correlation with the human panel is within 0.05 of the K-averaged teacher.

Ablations. Three ablations isolate the key ingredient:

  • Hard-label distillation from a single teacher sample, which tests whether averaging matters.
  • Soft targets without the metamorphic losses, which tests whether the property training matters.
  • A holistic score without atomic step verdicts, which tests whether decomposition matters.

Why not just…?

Why not just sample the frontier judge many times and average? This is the most accurate option and serves as our teacher. It fails the constraint that motivates the work: genomic data under privacy rules cannot leave the lab. K-fold sampling also multiplies cost. In this regime the experiment that matters is whether the student matches the averaged teacher, and we report the gap directly. If the gap is large, the honest conclusion is that local judging is not yet viable. Averaging also does not by itself fix monotonicity failures. If the teacher systematically misses a swapped-label bug, its average misses it too, and the bug-monotonicity tests will show this.

Why not just put a probe on a larger open model, with no distillation? This is a serious threat, because the latent-information paper shows probes already reduce instability. If a probe on an untuned 70B model matches our student on held-out tasks, then the distillation and metamorphic losses add little, and the remaining argument is only size. A 70B model is beyond typical lab hardware, and the BioAgents line targets roughly 3B models. We therefore report every method at matched parameter count and also against the 70B probe. A probe trained on the 7–8B base is the clean comparison.

Why not just check outputs deterministically against the right answer? On a benchmark this is better than any judge, and we use it to validate. The goal, though, is reference-free judging on a user's own dataset, where no answer key exists. That is exactly where an agent confidently analyzing corrupted data does harm. We test this by holding out ground truth and checking whether the judge flags the corrupted-input runs that deterministic checks catch only when the answer key is present.

Why not just ground the agent better with ontologies and retrieval, as BioAgents does? Grounding improves generation but leaves the evaluation problem untouched. BioAgents still failed on multi-step pipelines, and BioAgent Bench agents still accepted corrupted inputs. A better agent still needs a trustworthy local check.

Proposed Experiments

The primary test runs on held-out BioAgent Bench tasks. It compares five judges: the student, the frozen base, the single-sample and K-averaged teacher, and the open-model probe. It reports four quantities. The first is self-consistency alpha across five runs. The second is inter-scale consistency across binary, Likert and numeric formats. The third is pass rates on each metamorphic property. The fourth is Spearman correlation with the human panel, alongside calibration error of the expected score. The calibration check directly addresses the distortion concern raised by the latent-rating paper.

Decisive check. The proposal succeeds only if all three hold. The student's alpha exceeds the single-sample teacher's. Its specification-monotonicity pass rate exceeds the teacher's; Certifying the Judge found that property to be the strongest predictor of human agreement. And its human correlation stays within 0.05 of the averaged teacher. If invariance training raises consistency but lowers human correlation, we will have shown the student learned to ignore traces rather than to judge them.

Robustness check. The student is trained on BioAgent Bench workflows and then scored zero-shot on single-cell analyses from the CellAgent dataset collection, using CellAgent's annotation quality as an external signal. This tests transfer to a modality that is largely new to the student.

Downstream check. The student serves as a local selector over three runs of an open-weight agent, limited to three rounds given the self-reflection trap BioAgents reports. We measure whether completion on BioAgent Bench rises above the 82.5% single-run figure reported there for GLM-4.7. We also recompute benchmark completion rates with prediction-powered intervals, to show how much BioAgent Bench's agent ranking depends on which judge is used.