Systematic reviews in medical AI often cannot average results across papers, because the papers use different data, preprocessing, and metrics. Reviewers fall back on narrative synthesis instead: they sort findings into themes, weigh them, and write a reasoned verdict such as "3D models help for low-contrast organs." These verdicts are almost never checked against reality. This proposal checks them. It picks one contested question, when a model that sees the whole 3D scan beats a model that sees one 2D slice at a time, and asks whether a structured narrative synthesis of the published comparisons predicts what a controlled re-run shows.
Medical image segmentation is a rare field where this test is affordable. The underlying experiments use public scans and can be repeated under one fixed pipeline. The work therefore produces two things. The first is a calibration of narrative synthesis on a real, conflicted literature: how often its conclusions and certainty ratings hold up. The second is a measured account of when volumetric context actually matters.
Core Claim
The literature on volumetric context disagrees with itself. 3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes argues that 3D kernels, which slide over depth as well as height and width, are needed where single slices are ambiguous. The ImageCLEF tuberculosis overview (Overview of ImageCLEFtuberculosis 2019) frames slicing as a trade-off between compute and context. AMOS finds a plain 3D U-Net beating a 3D transformer on abdominal CT (89.04% versus 86.32% mean Dice, a measure of overlap between predicted and true masks). The biplanar X-ray benchmark (Benchmarking Encoder-Decoder Architectures for Biplanar X-ray to 3D Bone Shape Reconstruction) reports the reverse ordering for projection-to-volume mapping. It also finds that Dice can hide the loss of clinically critical landmarks.
These papers differ in task, preprocessing, slice spacing, and metric all at once. No reader can tell which difference drives which result. That tangle is exactly the situation narrative synthesis exists for.
Our claim is that a narrative synthesis's directional conclusions can be tested. Each conclusion has the form "architecture family A beats B on structure class C under metric M." We can rerun those conditions on public data with preprocessing, tuning budget, and parameter count held fixed. The agreement rate between synthesis and replication is then a direct, previously unmeasured estimate of how trustworthy this kind of synthesis is.
The umbrella review Artificial Intelligence Performance in Image-Based Cancer Identification motivates this directly. It rated 80.4% of reviews high quality, yet only 5% of the underlying evidence reached high certainty. That result says reviews may be well-conducted while resting on weak evidence. Nobody has measured whether their conclusions are right. That umbrella review synthesizes the literature but never re-executes anything, and that is the step we add.
The substantive hypothesis we expect the synthesis to surface, and which the replication tests, is specific. The gain from 3D context should grow with how fast a structure's cross-section changes between adjacent slices, and shrink as slices get thicker. We measure this "through-plane shape change" from the ground-truth masks alone, so no model is involved and the argument is not circular.
Research Program
Stage 1: the synthesis. We assemble a corpus of published head-to-head comparisons of slice-wise 2D, 2.5D (a few neighbouring slices stacked as input channels), 3D convolutional, and 3D transformer models on volumetric segmentation and report generation. We use the papers in this file as the seed set. Two coders, blind to Stage 2, extract each comparison's task, structure, modality, slice spacing, preprocessing choices, metric, and direction of result. They follow a pre-specified four-step protocol: question, evidence gathering, appraisal, synthesis. Findings are grouped into thematic categories: structure geometry, contrast, acquisition, pipeline, and metric family. The coders then write a line-of-argument synthesis, meaning a single explanatory account assembled from the individual findings.
The synthesis must end in a pre-registered set of directional claims. Each claim carries a GRADE-style certainty rating (the standard high/moderate/low/very-low grading of evidence strength) and a stated confidence.
Stage 2: the controlled re-execution. For every claim whose conditions can be reproduced on public data, we rerun the comparison in one shared codebase with these fixed elements:
- Resampling and intensity normalization: a single recipe applied to every run.
- Cropping: a single cropping policy.
- Parameter count: matched across architectures.
- Compute: equal GPU-hours for hyperparameter search.
- Seeds: five per configuration.
Slice spacing is varied deliberately by resampling, so that the synthesis's spacing claims can be tested. Each result is reported under both geometric metrics and clinical metrics. This follows the Dice-paradox finding in the biplanar X-ray benchmark.
Stage 3: the audit. Each synthesized claim is scored as confirmed, reversed, or null against the replication, with a replication counted as null if its 95% interval covers zero. We then measure two things:
- whether certainty ratings predict replication success;
- which categories of claims fail most often.
We expect claims about metric-dependent conclusions and pipeline-confounded conclusions to fail most.
Baselines
Vote counting. The naive reference point is the oldest shortcut in evidence synthesis. For each architecture pair and structure class, we take the direction reported by the majority of papers. It uses the same extracted corpus and costs almost nothing. The narrative synthesis beats it only if its agreement rate with the replication is higher, with a paired bootstrap over claims whose interval excludes zero.
Quantitative meta-regression. This is the strongest competitor on the question of synthesis. We regress the reported 3D-minus-2D Dice difference on the coded study features (structure geometry, spacing, modality, parameter count) and read off predicted directions. It is the method a statistician would pick, and it uses the same coded features. The narrative synthesis must match or beat its agreement rate. It should do especially well on claims involving non-Dice metrics, where meta-regression has too few comparable data points.
Controlled segmentation references. These check that our pipeline is not handicapping any architecture. On AMOS CT, our 3D U-Net should reproduce the reported 89.04% mean Dice from AMOS. A clear shortfall would mean our pipeline has introduced exactly the variance we are trying to remove. SegVol, a large 3D foundation model reporting 0.8593 Dice on AMOS22, is run frozen with its released weights as an upper reference for volumetric context. On biplanar reconstruction, the reference points are AttentionUNet at 93.66% and SwinUNETR at 93.64% femur Dice from the biplanar benchmark. On report generation from chest CT, the reference is CT2Rep. It reports a clinical-efficacy F1 of 0.534 for its 3D extractor against 0.456 for its 3D baseline. We add a slice-wise 2D encoder with the same decoder and parameter budget.
Ablations isolating the synthesis's key ingredient. The first removes pipeline and metric coding from the synthesis, so coders see only task, architecture, and result. This isolates whether explicitly tracking confounders is what makes the synthesis accurate. The second removes the line-of-argument step, leaving only thematic grouping. This tests whether building a cross-study explanation adds predictive value over grouping alone.
Why not just…?
Why not just run the controlled benchmark and skip the synthesis? For the handful of conditions we can afford to run, the benchmark is the better answer, and we treat it as ground truth. But most of the literature concerns private cohorts, proprietary scanners, and tasks nobody will rerun. Reviewers will keep synthesizing that literature narratively, as the umbrella review does over 158 reviews. The point is to learn how much those syntheses can be trusted. The only way to learn it is to test them where testing is possible.
Why not just do a quantitative meta-analysis? This is a real threat, and we run it as a baseline rather than argue it away. Reviews such as Decision aids on breast conserving surgery found pooling impossible under measurement heterogeneity. Here, heterogeneity is partly measurable (spacing, parameters), so meta-regression may do well on Dice-based claims. If it matches the narrative synthesis on every claim category, we will report that narrative synthesis adds nothing in this setting. That would be a useful negative result.
Why not just have a large language model write the synthesis? It is cheap and increasingly common. SpineBench and RadFM show general models underperform on fine-grained medical reasoning, but those results concern image interpretation, not literature synthesis. We therefore add an LLM-written synthesis arm. It receives the same corpus and protocol and is scored against the same replications. Whether it matches the human coders is an open empirical question this design answers.
Proposed Experiments and Decisive Checks
Data. Segmentation replications use AMOS CT and MRI, with scanner-held-out splits to probe the more-than-20% drop on unseen MRI scanners that AMOS reports. They also use the biplanar X-ray benchmark's four CT sources (TotalSegmentator, CTPelvic1k, VerSe2019, RSNACervicalFracture), including its 10-degree-tilt and fracture/implant splits. Report generation uses CT-RATE with clinical-efficacy F1. Anomaly localization, where the synthesis makes claims about volumetric context, uses MOOD. No reported result was found on Lacuna for MOOD, so our runs there establish their own references.
The through-plane hypothesis. We regress per-structure 3D-minus-2D gain on mask-derived through-plane shape change and on resampled slice thickness. The hypothesis passes if the shape-change coefficient is positive with its interval excluding zero across seeds and datasets. It fails if gains are flat or track parameter count instead.
Decisive checks. Three outcomes would settle the project's claims:
- Synthesis versus shortcuts. If the narrative synthesis's agreement with replication is no better than vote counting, formal narrative synthesis offers no epistemic gain on this literature.
- Calibration of certainty ratings. If GRADE-style certainty ratings do not rank claims by replication success, those ratings are uncalibrated here. That would bear directly on the evidence paradox the umbrella review documents.
- Metric dependence. If a claim's direction flips between Dice and clinical landmark metrics in replication, any synthesis that pooled across metric families is shown to be unsafe. We will count how many published conclusions carry this failure.
References
- 3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes
- Overview of ImageCLEFtuberculosis 2019 - Automatic CT-based Report Generation and Tuberculosis Severity Assessment
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Benchmarking Encoder-Decoder Architectures for Biplanar X-ray to 3D Bone Shape Reconstruction
- Artificial Intelligence Performance in Image-Based Cancer Identification: Umbrella Review of Systematic Reviews
- SegVol: Universal and Interactive Volumetric Medical Image Segmentation
- CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging
- Decision aids on breast conserving surgery for early stage breast cancer patients: a systematic review
- SpineBench: A Clinically Salient, Level-Aware Benchmark Powered by the SpineMed-450k Corpus
- Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical data