Concordance Checks for Grading Vessel Segmentations Against Incomplete Labels

Programs that trace lung blood vessels on CT scans are now good enough to find small branches that the human labels in public test sets never marked. Standard scoring counts every such branch as a mistake. The result is that the most anatomically complete model can look the least precise. This proposal borrows a single idea from uranium-lead dating of zircon crystals. Zircon dating trusts an age only when two independent radioactive clocks inside the same grain agree. The lung offers a similar pair of independent signals: each pulmonary artery runs beside a bronchus of matching size. We propose using artery–airway agreement, measured by two separately trained models, as an internal referee. The referee decides whether an "unlabeled" predicted artery branch is a real vessel or a leak into surrounding tissue. Combined with an explicit model of how much each benchmark's annotators left out, this gives a precision estimate that stays honest across benchmarks whose labeling protocols differ.

The stakes are practical. Preoperative vascular mapping, meaning patient-specific 3D maps of vessels built before surgery, matters because anatomy departs from the textbook often. One example is an aberrant left hepatic artery whose ligation would have destroyed blood supply to part of the liver (case report). If the benchmarks used to choose mapping models penalize completeness, the field will select models that miss exactly the small branches surgeons need.

The Problem in Detail

Two papers frame the gap. AirRC built 254 densely annotated CT scans covering pulmonary arteries, veins, airway lumen and airway wall. Its 3D U-Net reached internal Dice overlap of 0.950 for arteries and 0.953 for veins. Zero-shot, the same model scored only "> 0.80" on ATM'22, Parse2022 and HiPas. The authors attribute part of that gap to a "precision paradox": true distal branches that the external annotators omitted are scored as false positives. ATM'22 shows the second half of the problem. Leakage, meaning segmentation that spills into lung tissue, is a real failure mode. When models met COVID-19 ground-glass opacities, branch detection fell from about 90% to about 64%.

Any fix therefore has to do two things at once. It must stop penalizing real unlabeled branches, and it must keep penalizing leakage. Recall-style topology metrics such as ATM'22's Branch Detected and Tree Length Detected sidestep the first problem but ignore the second.

This is a study-heterogeneity problem in the clinical-evidence sense: different benchmarks embody different protocols, here different stopping depths for annotation. Pooling scores across them is unsafe for the same reason pooling trials with conflicting inclusion criteria is unsafe. CTKG (CTKG) and PyHealth (PyHealth) harmonize such differences at the level of vocabulary and task definitions. They motivate the framing but supply no method for imaging labels.

Core Claim

The proposal has two parts. The first models each benchmark's incompleteness explicitly as an annotation-completeness curve, the probability that a true branch of a given caliber and branching depth was labeled. The second adjudicates each unlabeled predicted artery branch with a concordance test against an independently segmented airway tree. Together they yield a precision estimate whose error against dense re-annotation is substantially smaller than that of naive precision or of masking predictions outside the labeled region.

Concretely, a predicted artery segment $s$ absent from the reference is scored concordant if an airway segment lies within distance $d$. That airway segment must also be roughly parallel (angle below $\theta$) and have a diameter ratio near the normal bronchoarterial ratio. The adjusted precision is

$$ \hat{P} = \frac{TP + C}{TP + FP}, $$

where $TP$ and $FP$ are the usual labeled-reference counts and $C$ is the number of reference-"false-positive" artery segments judged concordant.

Veins run between lung segments rather than beside bronchi, so they cannot be refereed this way. For veins we report only the completeness-curve correction, and we state this limitation openly.

The zircon analogy carries a warning as well as a method. Concordance means something only if the two clocks are independent and the system stayed closed. Here independence means the artery and airway models must not share errors. We train them separately, on different source datasets where possible. The "open-system" case is diseased lung, where consolidation can corrupt both models at once. That case is a predicted failure regime, not something to hide.

The completeness curve also borrows from detrital zircon work. Detrital age spectra are distorted by caliber-dependent sorting and must be corrected before populations are compared (Ordos Basin provenance; Songpan-Ganzi). In the same way, we fit a per-benchmark logistic curve of labeling probability against vessel radius and branching depth. The curve is fitted from the overlap between high-confidence predictions and the reference. This makes each benchmark's protocol a measured quantity rather than a hidden one.

Electrospinning enters only as a downstream consumer, and we keep its role small. Turning segmented trees into fabrication geometry for patient-specific electrospun grafts or wraps would require topologically faithful branches, a need the research file identifies as unmet. The proposal does not attempt that translation.

Research Program

We will first build the concordance referee on top of artery, vein and airway segmenters trained on AirRC. The artery and airway models are trained as separate networks with different random seeds and, in one variant, different training sources: Parse2022 for arteries and ATM'22 for airways. This lets us test how much shared training data inflates agreement.

We will then fit completeness curves for ATM'22, Parse2022 and HiPas. The analysis will report how each benchmark's labeling probability decays with vessel radius, turning the "precision paradox" into a quantified, per-dataset bias.

Finally, we will re-rank a set of segmentation models under each metric. The set includes AirRC-trained, Parse2022-trained, nnU-Net default and ATM'22-style topology-loss models. The question is whether the conclusion "which model is best for surgical mapping" changes.

Baselines

The naive reference point is standard voxel and segment precision against each benchmark's labels as released. For the strongest model, the only external figure in the file is AirRC's zero-shot Dice "> 0.80" on ATM'22, Parse2022 and HiPas. We found no reported external precision value on Lacuna for AirRC or for the ATM'22 winner. We will therefore recompute naive precision for every model ourselves under identical preprocessing, with all scans resampled to 1 mm isotropic.

The strongest existing evaluation alternative is the recall-oriented topology scoring of ATM'22: Tree Length Detected and Branch Detected, on which the winning "timi" pipeline reported 95.9% and 94.7%. These scores ignore false positives by design. We pair them with a masked precision that discards predictions lying beyond the distal endpoints of the labeled tree. This masked variant is the most direct competitor to our metric, because it also refuses to penalize unlabeled distal branches.

The key ablation removes the airway partner. It keeps the same geometric gates (distance, smoothness, caliber taper) applied to the artery prediction alone. If concordance with an independent structure matters, this shape-only referee should accept leaks that the full referee rejects. A second ablation trains artery and airway models on the same data and seed, to measure the cost of losing independence.

All metrics are computed on the same predictions, so compute and tuning budgets are equal. The thresholds $d$ and $\theta$, and the masking extent, are tuned only on AirRC training folds and then frozen. Our metric counts as better if its absolute error against dense-truth precision is lower than masked precision's by more than the bootstrap 95% interval across scans. It must also keep that advantage under injected leakage.

Why not just…?

Why not just densely re-annotate the external test sets? That is the right gold standard, and we use it on a subset. Dense 4-stage human-in-the-loop labeling of the kind AirRC needed is too costly to repeat for every new benchmark or model release. The proposal is worth doing only if a cheap metric reproduces what dense labeling would say. Experiment 3 measures exactly that.

Why not just mask out predictions beyond the labeled tree? This is a real threat, and it may work about as well on clean screening scans. Its weakness is that it gives a free pass to leakage occurring beyond the labeled extent. That is exactly where, per ATM'22, leaks concentrate in peripheral and diseased lung. Experiment 2 injects leaks at controlled depths. If masked precision tracks true precision as well as our metric does there too, we will drop the concordance claim and report the completeness curves alone.

Why not just report recall-style topology metrics and ignore precision? Branch Detected and Tree Length Detected cannot distinguish a complete model from one that floods lung tissue. A surgeon planning around an aberrant artery needs both properties. The ranking analysis will show whether recall-only scoring selects leaky models.

Why not just use model confidence or ensemble disagreement to flag doubtful branches? Confidence comes from the same network that produced the branch, so it shares that network's errors. That is the independence failure the dating analogy warns against. We include ensemble variance as an additional referee in Experiment 1 for comparison.

Proposed Experiments

The first experiment is controlled censoring on AirRC, using its five-fold splits. On held-out folds we truncate the dense reference at fixed branching depths or radius thresholds, which mimics the protocols of the external benchmarks while keeping the true answer known. We then measure how far naive precision, masked precision, the shape-only ablation and the concordance metric each fall from true precision.

The second experiment injects synthetic leakage on the same data. We graft false, tissue-following branches onto predictions at controlled depths and check that the concordance metric penalizes them while masked precision does not.

The third experiment is the decisive real-data check. A thoracic radiologist, blinded to model identity, densely completes the labels for a stratified sample of roughly twenty scans drawn from Parse2022 and HiPas. Every metric's error is then measured against this new truth.

The fourth experiment probes the open-system regime on the ATM'22 COVID-19 cases, where branch detection is known to collapse. Here we expect correlated failure of the artery and airway models. The metric should signal this by abstaining when the airway model's own confidence is low, rather than certifying false branches.

The proposal fails if masked precision matches concordance error within bootstrap intervals in both the leakage and real-data tests. It also fails if model rankings are unchanged across metrics, since then the precision paradox would be cosmetic rather than consequential.