Teaching the Coarse Stage to Predict What the Fine Stage Will Never See

Many medical segmentation systems work in two passes. A cheap first pass looks at a shrunken version of the whole scan and decides roughly where the organ or tumor is. An expensive second pass then looks at full resolution, but only inside the region the first pass selected. This saves a great deal of compute, but it has a failure mode that no amount of second-pass accuracy can fix. If the first pass misses part of a faint, low-contrast target, such as the thin tail of a pancreas, a tumor margin invading the chest wall, or a narrow esophagus, the second pass never sees that tissue. The result can be a 0% overlap score on exactly the hardest patients. The obvious remedy is to let the second pass look wider wherever the first pass is uncertain. It fails for a structural reason: when shrinking the image has erased a faint boundary, the first pass is confidently wrong rather than uncertain, so its uncertainty map is blank over the missed tissue.

We propose to give the first pass a small, separate output that predicts one thing: the probability that true target tissue lies at a location the second pass will not look at. This head is trained on the actual misses a cascade makes on held-out training cases, not on softmax entropy or synthetic noise. It also sees native-resolution intensity statistics that downsampling would otherwise destroy. At test time, a fixed budget of extra full-resolution windows is spent where this "omission risk" is highest. The claim is narrow and testable. At an equal number of voxels processed by the fine stage, omission-risk routing should cut catastrophic cascade failures more than simply enlarging the crop or routing on standard uncertainty.

Core Claim

The research file documents irreversible coarse-stage omission in both segmentation cascades and matching pipelines. A 3D Coarse-to-Fine Framework for Automatic Pancreas Segmentation runs a coarse 3D network to crop a bounding box and a fine network only inside it. The authors acknowledge 0% DSC cases on the JHMI pathological dataset. Uncertainty-Guided Coarse-to-Fine Tumor Segmentation with Anatomy-Aware Post-Processing is the closest existing work. It adds Monte Carlo dropout uncertainty, meaning uncertainty estimated from the spread of several randomized forward passes, but only in the second stage and only to reweight training losses. Its Top-1 component filter deletes secondary lesions in multifocal disease, and nothing flows back to change the crop. LoFTR: Detector-Free Local Feature Matching With Transformers describes the same trap for feature matching, where a correspondence missed at the coarse level cannot be recovered by the fine search. These papers motivate the proposal.

Other work constrains the design. Feng et al. point out that uncertainty computed on downsampled features can read as zero over thin structures that vanished during pooling. The resolution-bias estimator in Boosting the Dual-Stream Architecture in Ultra-High Resolution Segmentation with Resolution-Biased Uncertainty Estimation selects top-K uncertain patches using entropy plus high-frequency feature residuals. The file notes that such residuals may vanish in low-contrast CT, where high frequencies are weak even at native resolution. Gate to the Vessel: Residual Experts Restore What SAM Overlooks trains correction experts on randomly masked and jittered masks, and its authors say this recipe does not carry over to diffuse, low-contrast targets. Together these rule out two designs: routing on softmax uncertainty from the coarse stage, and training the router on synthetic perturbations. Our design replaces both with a supervised target built from real cascade errors, plus an input pathway that sees full-resolution contrast.

The difference from Isler et al. is concrete. They use uncertainty inside the crop to shape training. We predict risk outside the crop and use it at inference to decide where the fine stage looks.

Research Program

The system keeps a standard coarse network (a 3D U-Net-style model on a downsampled volume) and a standard fine network (the same architecture at native resolution on crops). We add three things.

The omission-risk head. This is a small extra output on the coarse network that predicts a voxel map $r(x)$. It is fed by two inputs: the coarse decoder features, and a lightweight stem that computes local intensity statistics at native resolution. These statistics are local variance and histograms within soft-tissue intensity windows, pooled only afterwards, so faint gradients survive into the risk head even though the segmentation pathway never saw them.

Supervision from real misses. We train with K-fold cross-fitting. The coarse and fine models are trained on K−1 folds, and the full cascade is run on the held-out fold. A voxel is labeled "omitted" if it is ground-truth foreground and the fine stage either never processed it or labeled it background. The risk head learns to predict these labels. The labels encode the structured, deterministic errors that downsampling and partial-volume blur actually produce, which answers the recipe problem raised by FineSAM++.

Budgeted routing. Given a fine-stage budget of $B$ voxels beyond the base crop $C_0$, we choose extra windows $W_1,\dots,W_m$ of fixed size to solve

$$\max_{W_1,\dots,W_m} \sum_{x \in \bigcup_j W_j \setminus C_0} r(x) \quad \text{subject to} \quad \Big|\bigcup_j W_j \setminus C_0\Big| \le B.$$

This objective sums predicted risk over newly covered voxels. It is monotone and submodular, so greedy selection is near-optimal. The same machinery covers both multifocal lesions and margins cropped out of the main box.

The fine stage also receives the coarse probability map as a prior, as in auto-context. We multiply that prior by $(1 - r)$ so the fine stage is not told "background" with full weight in regions the coarse stage is likely to have missed. The motivation is the concern, documented for 3D Enhanced Multi-scale Network for Thoracic Organs Segmentation and for restoration cascades in Deep Residual Network for Joint Demosaicing and Super-Resolution, that downstream stages treat upstream errors as truth. This gating is a secondary component and is ablated separately.

The analysis centers on a coverage-versus-compute curve. For each routing method, we sweep $B$ and plot two quantities: the fraction of ground-truth foreground ever processed by the fine stage, and final segmentation quality. The proposal succeeds if omission-risk routing sits on or above every other curve at all budgets. It does not need to win at a single operating point.

Baselines

Plain cascade. The naive reference point is the unmodified coarse-to-fine cascade with the bounding box from thresholded coarse output. We reimplement it on our splits. For context, A 3D Coarse-to-Fine Framework for Automatic Pancreas Segmentation reports 84.59% average DSC on NIH Pancreas, with worst-case DSC near 70%. On JHMI it reports 80.56% average DSC with acknowledged 0% cases. Because published splits and preprocessing may differ, we compare against our own rerun and cite these values only as a sanity check.

Margin dilation. The strongest simple competitor enlarges the coarse box by a margin, or lowers the coarse threshold, until the fine stage processes the same number of voxels as our method. This is the fairest test of whether learned risk adds anything beyond "look at more tissue." It is tuned over the same budget grid with the same validation data.

Uncertainty routing. This is the strongest competitor in the uncertainty direction. It spends the same extra-window budget using coarse-stage predictive entropy, and separately Monte Carlo dropout variance in the style of Uncertainty-Guided Coarse-to-Fine Tumor Segmentation with Anatomy-Aware Post-Processing. That paper reports that its two-stage refinement raised Dice from 0.4690 to 0.6447 and cut HD95 from 187.97 mm to 8.89 mm on Orlando Health lung CT. We cannot rerun that private dataset, so we implement its uncertainty signal on public data. We also run the resolution-biased estimator of Boosting the Dual-Stream Architecture in Ultra-High Resolution Segmentation with Resolution-Biased Uncertainty Estimation as a routing signal, adapted to 3D.

Fine-only reference. Full sliding-window inference at native resolution is the upper bound on coverage, with no omission possible. It shows how much of the full-resolution quality we recover at a fraction of its cost. Zhou et al. report that their cascade cut per-case time by 36% relative to fine-only, to 245 seconds.

Ablations. Four ablations isolate the key ingredients. We remove the native-resolution intensity stem. We replace real out-of-fold miss labels with FineSAM++-style synthetic mask perturbations. We route with random windows at equal budget. We remove the $(1-r)$ prior gating.

All methods share backbones, training data, augmentation, epochs and fine-stage voxel budget. Hyperparameters get the same number of validation trials. Beating a baseline means a statistically significant improvement, by paired bootstrap across test cases, in two measures at matched budget. The first is worst-decile DSC. The second is the count of cases below 50% DSC. Mean DSC must not decrease.

Why not just…?

Why not just enlarge the crop? This is the real threat. If faint misses cluster near the coarse box, a uniform margin captures them almost as well as learned risk. The matched-budget curve settles it directly. Where misses are spatially dispersed, as with multifocal lung lesions, disconnected pancreas tails, and cystic deformations on the pathological cohort, a uniform margin should waste budget on low-risk shell voxels. If dilation matches us across budgets, we will report that learned routing is unnecessary for these anatomies.

Why not just route on entropy or Monte Carlo dropout from the coarse stage? The prediction is that these signals are near zero on confidently missed tissue, as Feng et al. argue. We test this before building anything by measuring the AUROC of entropy, dropout variance and our risk head for classifying held-out omitted voxels against correctly rejected background. If entropy already reaches high AUROC on these voxels, the premise fails and the project reduces to entropy-based crop expansion.

Why not just skip the cascade and run a single native-resolution network? That removes omission but pays full compute. The file also reports that a single-stage full-volume Swin UNETR reached only 0.4690 Dice on Orlando Health because background overwhelmed it (Isler et al.). We include fine-only as a reference to quantify the accuracy-per-compute gap rather than assume it.

Why not just recycle predictions iteratively? Abstract: RecycleNet - Latent Feature Recycling Leads to Iterative Decision Refinement refines by feeding features back through the network. It refines what is already in view, so it cannot restore tissue outside the crop. It is complementary, and we leave it out of the main comparison.

Proposed Experiments

The primary benchmarks are NIH Pancreas and MSD Pancreas/Tumor, which are public and have low-contrast organ boundaries. On MSD Pancreas, DB-MSMUNet:Dual Branch Multi-scale Mamba UNet for Pancreatic CT Scans Segmentation reports 87.59% DSC. It reports 89.47% on NIH with a non-cascade dual-branch design. We do not claim to exceed those means, since our target is tail failures at matched compute.

SegTHOR's esophagus tests thin tubular omission. 3D Enhanced Multi-scale Network for Thoracic Organs Segmentation reports an esophagus DSC of 0.8597. NSCLC-Radiomics tests multifocal and invasive lung tumors. No reported result was found on Lacuna for any cascade on NSCLC-Radiomics, and we will report our own reruns. JHMI is used if access permits, since it holds the documented 0% failures.

Metrics fall into three groups. For quality, we report mean and worst-decile DSC, the count of cases below 50% DSC, HD95 and lesion-level recall. For coverage, we report the fraction of ground-truth foreground processed by the fine stage. For cost, we report fine-stage voxels and wall-clock time.

Three checks decide the project. The first is the AUROC test above, run first on NIH Pancreas with out-of-fold cascade outputs. The second is the matched-budget coverage curve against margin dilation. The third is the synthetic-versus-real miss-label ablation, which tests whether training on the cascade's own errors matters. If the risk head wins the first check but ties dilation on the second, the honest conclusion is that omission is predictable but spatially local. The contribution then becomes a diagnostic, not a router.