Certifying LLM Judges of Micro-CT Bone Reports with Controlled Volume Edits

Micro-computed tomography (micro-CT) scans a small specimen with X-rays from thousands of angles and rebuilds a 3D grid of density values at micron resolution. Biologists use it to measure trabecular bone, the sponge-like lattice inside bones, through numbers such as bone volume fraction, strut thickness, strut spacing and strut count. Increasingly they also describe what they see: "trabecular loss concentrated beneath the load-bearing surface, with disconnected struts consistent with osteolysis." Vision-language models will soon write such descriptions automatically. Nobody knows how to grade them. Voxel-overlap scores do not read text. Text-overlap scores reward fluent jargon. LLM judges, meaning large language models prompted to score other models' outputs, are known to reward length and confident tone.

We propose to certify LLM judges for micro-CT reports using ground truth built by construction. We take real micro-CT volumes and edit them in known, graded ways: we thin struts by morphological erosion, delete struts to break lattice connectivity, carve lesions at chosen locations, and inject acquisition artifacts that should not change the biology. The true answer to "what changed, where, and by how much" is therefore known exactly. A judge counts as trustworthy only if its scores respond to these edits monotonically and in the right direction, while staying flat under changes to wording, length, citations and scanner artifacts. The central question is whether a judge given a tool-computed evidence file (morphometry plus graph metrics of the lattice) can check the interpretive, spatial and topological claims that a simple rule-based numeric checker cannot. We design the study so that it can also show that the rule-based checker is enough.

Core Claim

Our claim is that judging micro-CT reports splits into two layers. The first layer holds quantitative claims ("bone volume fraction fell"). A deterministic checker handles these, and an LLM judge adds little there. The second layer holds relational, topological and inferential claims ("loss is subchondral," "connectivity is disrupted," "consistent with osteoporosis"). Here a judge grounded in an evidence file can be validated without continuous expert labeling, provided the validation uses falsifiable properties tied to physically controlled volume edits. The new ingredient is the edit-based certification. Existing judge stress tests perturb only the text being judged. Here the perturbation is applied to the scientific object itself, so monotonicity has a physical meaning: more simulated bone loss must make a "no change" report score lower.

Relation to Prior Work

The certification logic comes from CertJudge. That work validates judges of Lean formal code with properties it calls Identity, Bug Monotonicity, Spec Monotonicity and Stability. Its trust index reaches a Spearman correlation of up to 0.905 with expert labels, using a 75-pair human calibration set. We keep that logic and move the perturbation from program text to 3D bone volumes, where the degree of a "bug" is a continuous physical quantity rather than a discrete edit.

Two papers supply the list of failure modes to test. CALM catalogues verbosity, authority (fabricated citation), position and sentiment biases. Kahneman4Review finds that length explains about 74% of the score gap between AI-written and human-written reviews, and that judges reward fluent analysis over real fault-finding. These results constrain the design. We cannot hand an off-the-shelf judge a report and a rendering and trust the output.

The choice of evidence file comes from the micro-CT side. Hip osteoarthritis network analysis represents the trabecular lattice as a graph, with junctions as nodes and struts as edges. Graph features reach 96.5% osteoarthritis classification accuracy, against 74.6% for a CNN on the raw micro-CT. We read this as evidence that connectivity information is missing from voxel-level views, so the judge's evidence file must contain graph metrics and not only standard morphometry.

Several existing systems are adjacent but do not do what we propose. OmniCT and CT-RATE/CT-CHAT build vision-language models for clinical CT. They evaluate with multiple-choice accuracy or BLEU/METEOR/CIDEr, work at clinical rather than micron resolution, and do not study judges. VISTA3D and the CBCT tooth/bone system evaluate segmentation with Dice alone. SRRD regresses trabecular parameters from clinical CT and evaluates with intraclass correlation. These systems produce numbers that our evidence file would contain. None of them evaluates language about bone. The long-form medical QA benchmark is the closest existing study of medical LLM judges. It is text-only, and its judges needed answer-order swapping to control position bias. We inherit that swapping protocol.

Research Program

Source volumes. We start from micro-CT volumes of trabecular bone. The human femoral head set from the hip osteoarthritis study (7 patients, 7 cadaveric controls) is one source, if it can be obtained. The murine CT volumes in the VISTA3D suite are another. We add rodent femur and vertebra scans, which are routine in osteoporosis work such as the Dolichos lablab study and the You-Gui-Yin study. Each volume is cropped into regions of interest to stay within the memory limits noted for multi-gigabyte reconstructions.

Edits. Each source volume receives edits in four families, each at graded strengths.

  • Thinning uses erosion by known voxel radii. It shifts strut thickness and bone volume fraction predictably.
  • Disconnection removes a known fraction of graph edges within a chosen region. It lowers strut count and graph connectivity while barely changing bone volume fraction. This is the case where purely volumetric reasoning should fail.
  • Focal lesions carve osteolytic-like cavities at specified anatomical positions.
  • Nuisance edits add noise, partial-volume blur and beam-hardening streaks. These leave the biology unchanged.

We recompute standard morphometry and trabecular graph metrics on every edited volume. These values form the evidence file.

Candidate reports. For each (original, edited) pair we produce reports in several forms:

  • correct reports;
  • reports with the direction of change flipped;
  • reports with the region wrong;
  • overclaiming reports that state a diagnosis the change does not support;
  • reports that ignore disconnection;
  • reports that describe an artifact as pathology;
  • stylistic variants of each (padded length, invented citations, confident versus hedged tone, reworded).

Reports come from templated generation, from a general vision-language model given renderings, and from human writers for a subset.

Judges and properties. The judges are frontier LLMs in three conditions: rendered slices only; slices plus the evidence file; and slices plus the evidence file with a claim-by-claim checklist prompt in the style of dynamic checklist judging. We test five properties:

  • Identity: same input, same score, across repeated runs and answer orders.
  • Edit monotonicity: for a fixed incorrect "no change" report, the score must fall as edit strength rises.
  • Localization sensitivity: a wrong-region report must score below the correct one.
  • Nuisance invariance: artifact-only edits must not raise scores for pathology claims.
  • Style invariance: length, citation and tone variants must not move scores.

Following CertJudge, violations are combined into a variance-aware trust index. The index is calibrated once against a panel of two or three bone-imaging experts on roughly 75 report pairs.

Baselines

Voxel and parameter metrics (naive reference). Dice between segmentations, plus intraclass correlation between reported and computed morphometry. These cannot score free text at all. We use them to show which report errors are invisible without language evaluation, and to anchor the evidence file.

Text-overlap scoring. BLEU and METEOR against an expert-written reference report, as used for CT-CHAT. It runs on the same report pairs and gets no tuning. It is beaten if our judge's correlation with experts is higher by a margin outside bootstrap confidence intervals.

Rule-based claim checker (strongest non-LLM competitor). A parser extracts quantitative and directional claims (parameter, direction, rough magnitude, named region) from each report and compares them with the evidence file using fixed tolerances. It is the strongest competitor because most micro-CT reporting is quantitative, and a deterministic checker has perfect Identity and style invariance by construction. We give it the same evidence file and tune its tolerances on the same 75-pair calibration set used for the trust index. No result was found on Lacuna for any rule-based or LLM-based checker of micro-CT reports. The only nearby figures are CertJudge's Spearman of 0.905 in Lean code, and AgentEval's failure-detection recall of 0.89 for step-level judging against 0.41 for end-to-end outcome checks (AgentEval). Neither is directly comparable. Beating the rule-based checker means higher expert agreement, and higher recall of injected errors, on the relational, topological and inferential claim strata. On purely quantitative claims we expect a tie and will report it as one.

Off-the-shelf LLM judge. A frontier model given slices and the report, with a standard rubric and order swapping as in the medical QA benchmark. This is the "just ask GPT-4o" reference point. It uses the same model, the same prompt budget, and the same number of samples per item as our judge.

Ablations. Two ablations isolate what matters. The first removes graph metrics from the evidence file. It should hurt mainly on disconnection edits, if the hip osteoarthritis result carries over to judging. The second runs the evidence-file judge without edit-based certification, selecting and prompting it using expert labels only. Comparing its expert agreement with the certified judge's shows whether certification tracks human judgment or only adds robustness.

Why not just…?

Why not just check numbers with rules? This is the real threat, and the design addresses it head on. If most errors in real reports are wrong directions or magnitudes of standard parameters, the rule-based checker will match the LLM judge, cost nothing, and never drift. The claim strata settle it. If the LLM judge gives no gain on region, connectivity and overclaiming errors, the conclusion is that micro-CT reporting should be checked with rules, and the edit suite remains a useful benchmark for those rules. We regard that as a legitimate outcome, not a failure.

Why not just use expert annotators? Experts remain the calibration anchor. The graded edit suite, however, involves thousands of report-volume pairs, and every change to the report generator would need relabeling. The file records the cost of manual bone delineation at 147–160 minutes per case, against 17 seconds automated (CBCT system). CertJudge shows that a one-time set of 75 pairs can stand in for continuous relabeling. We test whether that holds here by checking the trust index against a held-out expert-labeled set.

Why not trust a strong LLM judge with good prompting? CALM and Kahneman4Review show that fluent length and invented authority move scores. The style-invariance and nuisance-invariance properties measure exactly those effects on bone reports. If the off-the-shelf judge already passes all five properties, the evidence file and certification are unnecessary, and the results will show that directly.

Proposed Experiments and Decisive Checks

The main experiment compares all judges on the edit suite. We report three things: rates of monotonicity and invariance violations by edit family; recall of injected errors by claim stratum; and Spearman correlation between the trust index and expert scores on held-out pairs.

Three results would be decisive.

  1. On disconnection edits, the evidence-file judge with graph metrics should detect "connectivity preserved" errors that both the rule-based checker without topological rules and the slice-only judge miss.
  2. Under length and citation padding, the certified judge's score shifts should be near zero, whereas we expect the off-the-shelf judge to show the verbosity effect CALM reports.
  3. On the human-written subset, whose error types were never injected, the trust index should still rank judges in the same order as experts.

If the third check fails, edit-based certification measures robustness to our own edits rather than trustworthiness, and we will say so.