Time-of-flight MR angiography (TOF-MRA) is a non-contrast brain vessel scan that makes blood bright only because fresh blood keeps flowing into the imaged volume. Slow flow, reversed flow, and turbulent flow all make a vessel that is physically open look faint, broken, or missing. Every public vessel-segmentation benchmark built on TOF-MRA therefore contains many small places where an annotator had to guess whether a dark stretch is a real absence or a flow artifact. Those guesses were made by individual readers with no shared rules, and they now serve as the ground truth that segmentation models are trained and ranked against.
We propose to write those rules down with a Delphi process, a structured, anonymous, multi-round expert survey. What makes this more than another consensus exercise is that the resulting rubric can be checked. The TopCoW benchmark pairs each TOF-MRA with a CT angiogram (CTA) of the same patient, and CTA depends on injected contrast rather than on flow. We can therefore test whether readers who apply the rubric to TOF-MRA alone reach the verdict that CTA supports. Delphi outputs are almost never validated against an independent physical reference. Here that validation is possible, and it decides whether the consensus is worth anything.
Core Claim
A small set of explicit, conditional criteria for classifying each gap in a Circle of Willis vessel segment can be produced by a hybrid human–AI Delphi panel of neuroradiologists and MR physicists. The criteria cover signal behavior at the gap ends, the calibers of neighboring vessels, how plausible collateral flow is, and the scan parameters. A gap is sorted into one of three classes: open but flow-attenuated, truly absent or aplastic, or indeterminate. We claim that non-expert readers applying these criteria to TOF-MRA will agree with CTA-supported verdicts more often, and with each other more consistently, than the same number of equally trained readers voting without the rubric. We further claim that scoring segmentation models with gap-class-aware credit changes their rankings on benchmark data in ways that plain overlap scores hide.
The novelty is not "Delphi for imaging." Metrics reloaded already used a 73-expert Delphi, reaching 93% consensus, to standardize validation metrics. Its scope, however, explicitly excludes physical artifacts, and it treats reference masks as fixed. The novelty is a consensus rubric aimed at the one place where the reference mask is itself uncertain, with an external oracle that can falsify it.
Research Program
Prior work plays three roles here.
The first role is motivation: several papers show the gap is real.
- Descoteaux et al. found that more than 25% of vessels visible on proton-density MRI were undetected on TOF in a single subject. Their method is an algorithm, and it offers no rule for deciding a given gap.
- Case reports on subclavian steal (Mair; Takahara et al.) and on slow-draining venous anomalies (Kim) show open vessels vanishing entirely on TOF.
- COSTA reports that older public labels missed low-flow vessels. It repaired them by individual radiologist refinement, without a shared protocol or any agreement statistic.
- TopCoW and the centerline-graph dataset fixed broken segments with VR-assisted single-annotator correction and A* pathfinding. They also showed that MRA gives systematically larger radii than CTA, so CTA is informative but not interchangeable with MRA.
The second role is supplying method. The Human-AI Hybrid Delphi uses a large language model as a literature-grounded "evidence floor" alongside about six senior experts. It reports 95% directional agreement with senior clinicians and a four-tier consensus scale that keeps "it depends" answers. We reuse this design, and we test one of its limitations directly: the paper reports that the model lacks experiential judgment, and bedside reading of flow voids is exactly that kind of judgment. We also borrow standard Delphi mechanics: anonymous rating, median and interquartile feedback, and written justification from outliers (behavioral trials Delphi). Item retention uses content validity indices.
The third role is constraint. The CTA–MRA radius discrepancy means CTA is used only as a tie-breaker where it is unambiguous. Gaps where CTA itself is unclear are excluded from accuracy scoring and kept only for agreement analysis.
What gets built. 1. Candidate gaps. We first extract candidate gaps automatically from the TopCoW MRA labels and centerline graphs. A candidate is any expected Circle of Willis segment (for example the communicating arteries, A1, or P1) that is missing, discontinuous, or below a caliber threshold on TOF. 2. Rubric development. The data are split by patient. On a development split, the panel generates candidate criteria from the physics literature and from TOF-only case review. They rate the criteria over two or three rounds. 3. Protecting the test set. The panel never sees CTA or any test-split case. 4. The rubric. The output is a written rubric with a decision procedure over the three gap classes. Each criterion is tagged with its consensus tier. 5. Gap-aware scoring. A second, shorter Delphi round settles scoring policy. For each class, it decides whether bridging the gap should be rewarded, penalized, or ignored. This supplies the artifact-specific extension that Metrics Reloaded left open.
Baselines
Naive reference. The unmodified TopCoW TOF-MRA labels, read as a gap verdict: present in the label means open, absent means absent. This is what current training and ranking assume.
Strongest human competitor. Independent multi-reader majority vote. This is the same number of readers, with the same training time and the same TOF-only cases, but no rubric. If structured consensus adds nothing beyond pooling judgments, this baseline will match us.
Strongest algorithmic competitor. The A* reconnection of Musio et al., used as a classifier: a gap it bridges counts as open. That paper reports a micro-averaged anatomical-variant F1 of 1.0 and node localization error under 0.25 mm. Those figures measure graph extraction against its own curated labels, not against CTA-verified gap status. For accuracy on CTA-verified gaps, no reported result was found on Lacuna, so we will run it ourselves on identical cases.
AI-only panel. An LLM-only panel following the HAH-Delphi evidence-floor setup. It rates the same criteria and classifies text descriptions of each gap.
Ablations. Two ablations isolate the key ingredients.
- A single-round rubric, with no feedback rounds, tests whether iteration matters.
- A rubric stripped of its conditional ("it depends") criteria tests whether nuance beyond a flat checklist matters.
What counts as beating a baseline. The rubric wins if it shows higher balanced accuracy against CTA-supported verdicts than majority vote and A*. The paired difference must have a confidence interval excluding zero, measured on the held-out patients. Inter-reader Fleiss' kappa must also be higher.
Why not just…?
Why not just use the paired CTA as the label and skip the panel? This is the real threat for TopCoW itself. Where CTA exists and is clear, it should win, and we use it exactly that way, as the oracle. But COSTA, OASIS-3, IXI and most clinical TOF scans have no CTA. CTA also measures different radii (Musio et al.) and can fail under abnormal circulation (Ying et al.). The rubric is how the CTA-checkable signal transfers to scans that lack CTA. If readers using the rubric do no better than chance at matching CTA, the project fails and we will say so.
Why not just have several radiologists vote? Voting may perform as well, and the head-to-head comparison against majority vote is designed to reveal this. If it ties on accuracy, the remaining value lies in a written, reusable decision procedure and in higher cross-dataset agreement on COSTA. We would report that more modest claim rather than claim superiority.
Why not just let an algorithm reconnect the gaps? Reconnection needs a fixed anatomical template. It will bridge truly aplastic segments, which is costly because a complete textbook Circle of Willis occurs in only 13–18% of patients (Musio et al.). We expect A* to over-call "open" on CTA-verified absences. We will measure its false-open rate directly.
Proposed Experiments
Experiment 1: validation against CTA. On held-out TopCoW patients, readers who did not sit on the panel classify every candidate gap under each arm. Arms are counterbalanced so that no reader sees the same case twice. We report balanced accuracy against CTA verdicts, false-open and false-absent rates by segment type, and Fleiss' kappa.
Experiment 2: transfer. The rubric is applied to a sample of COSTA scans from several vendors, where no CTA exists. There we measure only inter-reader agreement, and we check whether agreement holds across vendors.
Experiment 3: effect on model rankings. We rescore publicly reported TopCoW-style segmentation models, or nnU-Net-type models we train on the same splits, under the gap-aware scoring policy. We then measure how many rank positions change relative to Dice and clDice. TopCoW's top models exceed 90% Dice and 70% balanced accuracy on variants, so this experiment asks whether those numbers hide systematic gap errors.
Decisive check. The whole proposal rests on one comparison: rubric versus majority vote on CTA-verified gaps. A clear win justifies consensus-derived references for flow-attenuated vessels. A tie reduces the contribution to a documented protocol. A loss means expert rules encode shared misconceptions about flow physics, which would itself be worth publishing.
References
- Metrics reloaded: recommendations for image analysis validation
- Geometric Flows for Segmenting Vasculature in MRI: Theory and Validation
- Lack of flow on time-of-flight MR angiography does not always indicate occlusion
- A Case of Pontine Infarction due to Subclavian Steal Phenomenon Enhanced by an Arteriovenous Shunt for Hemodialysis
- Image Findings in Brain Developmental Venous Anomalies
- COSTA: A Multi-Center TOF-MRA Dataset and a Style Self-Consistency Network for Cerebrovascular Segmentation
- Benchmarking the CoW with the TopCoW Challenge: Topology-Aware Anatomical Segmentation of the Circle of Willis for CTA and MRA
- Circle of Willis centerline graphs: A dataset and baseline algorithm
- The Human-AI Hybrid Delphi Model: A Structured Framework for Context-Rich, Expert Consensus in Complex Domains
- An international, Delphi consensus study to identify priorities for methodological research in behavioral trials in health research
- Pitfalls of computed tomography angiography examination in veno-arterial extracorporeal membrane oxygenation patients: a case report of a patient with cardiac rupture