When the Genome Breaks the Template: Measuring Canonical-Prior Hallucination in LLM Descriptions of Mitogenomes

Ask a large language model to describe the mitochondrial genome of an obscure shrimp or rockfish, and it will produce a confident paragraph: 37 genes, a familiar gene order, a control region of some length. The mitochondrial genome (mitogenome) is the small circular chromosome inside mitochondria. Most animal mitogenomes share a conserved "textbook" layout, but many lineages depart from it. They move transfer-RNA genes, duplicate regions, or carry truncated tRNAs. The risk is that the model recites the textbook layout when the real genome differs. This is the biological version of what VALOR-EVAL called associative bias in image captioning: generating features that usually co-occur rather than features that are actually present.

Mitogenomes are an unusually clean place to measure this. Thousands of species have complete, annotated mitogenome records in GenBank, which lists every gene's coordinates, strand, and start and stop codons. So nearly every factual claim a model makes can be checked mechanically. We propose a benchmark and an evaluator built around that fact. The benchmark turns free-form LLM descriptions into typed atomic claims, meaning single checkable statements such as "trnK lies directly upstream of trnD." A symbolic checker verifies each claim against the GenBank record while respecting the fact that the genome is a circle. The benchmark then asks whether errors disproportionately take the shape of the clade's canonical template. The second question is less obvious and possibly more important: do generic LLM judges share the same prior, and wave those errors through?

Core Claim

We make two testable claims. The first is about generators. On species whose mitogenome departs from their clade's consensus layout, a large share of LLM factual errors will be "canonical intrusions." These are claims that would be true of the consensus layout but are false for this species. The effect should grow as the species becomes rarer in the literature. This extends the rarity penalty that FActScore documented for biographies into a domain where the "typical" answer is precisely specified and checkable.

The second claim is about judges. An LLM judge that sees the reference feature table, as in the setup of Evaluating Open-QA Evaluation, will falsely accept canonical intrusions at a markedly higher rate than it falsely accepts other errors. The reason is that its own prior agrees with the wrong claim. A typed symbolic checker in the spirit of MARJ from UGPhysics removes this bias for structural claims, and it does so without losing agreement on the prose claims it passes to an LLM.

We define canonical intrusion precisely. Let $O_s$ be the reference gene arrangement of species $s$ and $C_s$ the consensus arrangement of its clade, restricted to species with $O_s \neq C_s$. The intrusion rate is

$$ \text{IR} = \frac{\#\{c \text{ false under } O_s,\ \text{true under } C_s\}}{\#\{c \text{ false under } O_s\}}. $$

Here $c$ ranges over a model's structural claims. We compare IR with the rate expected if errors were spread over all arrangements reachable by one rearrangement event. The null is a matched random-error distribution, not zero.

Research Program

Building the reference set. We harvest complete annotated animal mitogenomes from GenBank. This follows the public-data recycling practice that already dominates the field, from the Luhua chicken to Sebastiscus rockfishes. We stratify along two axes. The first is whether the species' arrangement matches its clade consensus. Anchor cases include the trnK–trnD versus trnD–trnK transposition in Palaemon (OQ512152, OP650931) from the Palaemon comparison, and the control-region repeats in Sebastiscus. The second axis is rarity, measured by the count of GenBank records and literature mentions per species.

Automated annotation is itself error-prone. Automated Annotation of Mitochondrial Genomes documents misidentified start codons and missed truncated tRNAs such as trnS1 lacking the DHU arm (one of the tRNA's structural loops). We therefore keep a curated subset in which rearrangements are confirmed by the source publication, and we treat checker–record disagreements on that subset as items for expert audit, not as automatic ground truth.

Claim schema and checker. Each generated description is split into atomic claims, following FActScore's decomposition. Each claim is typed as gene content, adjacency or order, strand, length, start or stop codon, or untyped prose (for example, phylogenetic interpretations). Typed claims are verified by a deterministic checker. Order claims are compared on the circular, strand-aware adjacency graph. A description that starts at the control region and one that starts at COX1 are therefore equivalent. Length claims use a stated tolerance, and codon claims use the lineage's mitochondrial genetic code. Only untyped claims go to an LLM judge, which gets the flatfile and source abstract as evidence.

Generation protocol. We prompt several open and closed models for free-form descriptions, and separately ask them constrained questions such as "list the gene order starting at COX1." Comparing the two formats is necessary for the "why not just" questions below.

Human labels. Two annotators with mitogenomics training label a stratified sample of claims, giving roughly several thousand claims in total. We report Cohen's kappa. The meta-evaluation reference point is the human agreement EVOUNA reports, kappa between 86.4 and 100.

Baselines

The naive reference point is lexical matching of generated text against the flatfile, using exact match and token F1 on gene names, orders, and numbers. We include it because it is what a practitioner would write first. EVOUNA reports that exact match underrated DPR+FiD at 59.2 versus 68.9 under human judgment on Natural Questions. We also include BERT-Score, which EVOUNA found worst among its evaluators. Both run on identical claims and descriptions.

The strongest existing competitor is a generic LLM judge given the full GenBank feature table in context and asked to label each atomic claim. This is the EVOUNA-style setup with GPT-4-class models, which reached 96–98% accuracy/F1 against humans on TriviaQA. It is strongest because modern context windows fit an entire mitogenome feature table, so "no evidence" is no longer an excuse. We run it with the same claims, the same evidence, and a matched prompt-tuning budget, tuning on a development split that is disjoint from test species.

A second strong competitor is FActScore's pipeline with the retrieval corpus swapped from Wikipedia to chunked GenBank flatfiles and source abstracts. FActScore reports under 2% error against humans on biographies. For mitogenomes, no reported result was found on Lacuna for this or for any claim-level verifier. The same is true for the MARJ-style design, whose 98% human alignment is on physics only.

The key ablation removes circular and strand normalization from our checker while keeping everything else. This isolates whether rotation invariance matters or whether typing alone does the work.

"Beating" the strongest baseline means two things. Our evaluator's overall agreement with experts must be at least equal to the in-context LLM judge's. Its false-acceptance rate on canonical-intrusion errors must also be lower, with a confidence interval that excludes the judge's rate. The second condition is the one that matters.

Why not just…?

Why not just ask constrained questions and score them by exact match? For gene order, this may genuinely suffice, and we treat it as a real threat. If a model is asked for a COX1-anchored order in a fixed format, a string comparison after rotation catches most structural errors without any LLM. We run exactly this, and if its intrusion rates match the free-form measurements, we will report that the benchmark reduces to constrained probes for structural facts. We expect a gap, because free-form descriptions are what users receive, and FActScore found that errors concentrate later in long generations, where constrained probes never reach. Constrained probing also cannot score untyped interpretive claims, such as whether a rearrangement supports a sister-clade relationship.

Why not just put the feature table in a strong LLM judge's context? This is the main competitor, and the proposal succeeds or fails against it. EVOUNA itself flags overgeneralization as a judge failure mode: judges accept a broadly true claim where specificity is required. A canonical intrusion is exactly a claim that is true of the genus in general but false for this species. If the in-context judge shows no elevated false acceptance on intrusions, the symbolic layer is unnecessary for accuracy, and we would say so. The contribution would then shrink to the generator-side measurement plus a cost argument.

Why not just run FActScore on GenBank text? Retrieval returns chunks, and adjacency claims about a circular genome depend on the two ends of the linear record that a chunk may split. This is a hypothesis, not an established result. The flatfile-FActScore baseline and the circular-normalization ablation together test it directly.

Why not just use MITOS or tRNAscan-SE and skip the LLM entirely? For annotating new assemblies, deterministic tools remain the right choice, as NJGPT concluded for trees after direct LLM tree output hallucinated. Our target is different: LLMs already write summaries, reviews, and explanations of these genomes, and those texts need auditing. The deterministic tools supply our ground truth. They do not measure what the language model says.

Proposed Experiments and Decisive Checks

The first decisive check is the generator effect. On curated rearranged species, is IR significantly above the matched random-rearrangement null, and does it increase as rarity decreases? Controls are canonical-layout species of matched rarity, where overall precision should be higher. If IR sits at the null, the associative-bias story is wrong, and the benchmark reverts to a plain domain factuality test.

The second check is the judge effect. We compare the false-acceptance rate of each baseline judge on intrusion errors versus non-intrusion errors, with expert labels as truth. The proposal predicts a gap for LLM judges and none for the typed checker.

The third check is the ablation. Removing circular normalization should raise false rejections on correct order claims that begin at a different anchor gene. If it does not, rotation invariance is a non-issue in practice.

The fourth check is prompting mitigations. We test whether reflection-style prompting, which raised guideline consistency to 62.9% overall in the ROT study, reduces IR. We also test indirect cross-examination, where the model is asked about a gene's neighbors repeatedly and independently, following Do Language Models Know When They're Hallucinating References?, and check whether its inconsistency predicts intrusions better than the model's direct confirmation.

We report per-claim-type precision, IR with bootstrap intervals, evaluator–expert kappa, and cost per thousand claims. The benchmark and checker are released, with source-publication citations attached to every curated rearrangement.