Where Does a Slice Need Its Neighbors? Context Information Gain for Budgeted 3D Segmentation

Computers often read a CT or MRI scan the way a person reads a stack of photographs, one flat slice at a time. This keeps memory and compute low. The cost is that the model can no longer see how a structure continues from slice to slice, and the result is jagged, discontinuous organ outlines. The alternative is to run a fully 3D network everywhere, which is expensive. Neither choice asks the question that actually matters: at which locations does seeing the neighboring slices change the answer? We propose to measure exactly that. For each region of a scan, the target quantity is how much the model's uncertainty about the correct label falls when neighboring slices are added. We then train a cheap predictor of this "context information gain" from the 2D input alone and use it to send only the regions that need 3D context to an expensive 3D model.

The idea is borrowed from how structural biologists estimate conformational entropy, the thermodynamic "wiggle room" of a molecule. The Rotamer Density Estimator learns a probability density over side-chain angles, the rotations of the flexible branches on each amino acid, conditioned on the surrounding 3D structure. Its signal about binding comes from the difference in entropy between two contexts, bound and unbound, not from absolute entropy. We apply the same move to medical volumes. The quantity worth estimating is not how uncertain a 2D prediction is. It is how much of that uncertainty volumetric context removes. The two can diverge sharply: some regions are ambiguous no matter how much context the model sees.

Core Claim

Let $X_v$ be the in-plane evidence around voxel $v$, $C_v$ the additional through-plane context (the neighboring slices), and $Y_v$ its label. We define the context information gain as

$$G(v) = \mathbb{E}\left[\ell_{2D}(v) - \ell_{3D}(v)\right] \approx I(Y_v; C_v \mid X_v),$$

where $\ell_{2D}$ and $\ell_{3D}$ are the per-voxel log losses of a slice model and a slab or 3D model against ground truth. The approximation holds when both models are well calibrated, because expected log loss then equals conditional entropy. The difference of two conditional entropies is then the conditional mutual information between the label and the extra context: how much the neighbors tell you about the label beyond what the slice already shows.

Conditioning on more information never raises entropy, so $G \ge 0$ in expectation. The open question is where $G$ is large. Our claim is that $G$ is spatially sparse and structured. We expect it to concentrate on small tubular or elongated structures and on organ ends where cross-sections change quickly. We expect it to be small both on easy large organs and on regions that are ambiguous for intrinsic reasons, such as pancreas borders in patients with little visceral fat. Roth et al. found that post-hoc 3D smoothing could not fix those borders (Deep convolutional networks for pancreas segmentation in CT imaging).

If this holds, routing by predicted $G$ should beat routing by 2D predictive uncertainty at the same compute budget. The reason is that uncertainty-based routing wastes 3D compute on regions that stay ambiguous even with full context.

This also offers a way to reconcile two findings in the file. CARVE reports that, on average, spending budget on in-plane resolution beats denser slice sampling. Roth et al. show that dropping depth context is severe for specific organs. $G$ is the local quantity that could explain both: depth matters a great deal in a small fraction of places and very little elsewhere.

Research Program

We first train three segmentation models on AMOS CT (AMOS benchmark). The first is a 2D slice model. The second is a slab model that sees $\pm k$ neighboring slices. The third is a full 3D model, 3D UX-Net, which uses large depth-wise 3D kernels and reaches 0.900 Dice on AMOS after FLARE pretraining but costs 639.4G FLOPs. All three are temperature-calibrated on a held-out split so that differences in log loss are meaningful as differences in entropy.

On training volumes we compute the empirical gain map from the log-loss difference. We then fit a small regression head on the frozen 2D model's features that predicts patch-averaged $G$ from the 2D input alone. Regressing on noisy per-voxel differences still targets their conditional mean, which is the quantity we want.

At test time the 2D model runs everywhere. The gain head ranks patches, and the top fraction $\beta$ of patches is re-segmented by the 3D model and blended back. Sweeping $\beta$ from 0 to 1 traces a compute-versus-accuracy curve that runs from pure 2D to pure 3D.

The analysis has three parts. First, we map where $G$ concentrates, broken down by organ, by distance to organ extremities, and by slice thickness. For the thickness analysis we resample AMOS volumes to simulate clinical 3–5 mm slice spacing. Second, we compare $G$ with 2D predictive entropy, estimated both from calibrated softmax and from Monte Carlo dropout (running the model repeatedly with random units switched off). The point is to test whether they disagree and where. Third, we test whether the gain head keeps its ranking quality on AMOS MRI from unseen scanners, where AMOS reports drops of more than 20% mDice.

As a short check in the source domain, we will truncate the structural neighborhood given to the Rotamer Density Estimator and measure how its rotamer entropy estimates rise and how its SKEMPI2 correlation changes. The gain concept predicts that the damage concentrates at residues with high context gain. This check is secondary and is not required for the main claim.

Baselines

The naive reference point is the calibrated 2D slice model alone ($\beta=0$). The strongest classical cheap fix is the same model followed by Roth et al.'s post-hoc 3D Gaussian smoothing of the stacked probabilities, which raised pancreas Dice from 56% to 68% in their CT study. We tune the kernel width on validation data.

The strongest existing accuracy reference is full 3D UX-Net ($\beta=1$). The file reports 0.900 Dice on AMOS for it (FLARE-pretrained, fine-tuned). The AMOS paper itself reports 89.04% mDice on the CT test set for 3D-UNet. We retrain 3D UX-Net on our exact splits rather than quoting these numbers, because the proposal is judged against its curve.

The strongest efficiency competitor is Swin-UMamba†, an efficient state-space U-Net at 28M parameters and 18.9G FLOPs. It is the natural cheap native-context model. We will run it on the same AMOS split, retrained, since no AMOS figure is reported for it.

The key ablation replaces the gain head with routing by 2D predictive entropy, and separately by Monte Carlo dropout variance. Both use the same 3D model and the same $\beta$. A second control routes random patches at the same budget. An oracle routes by the true $G$, which bounds what any router could achieve.

All models share the same data splits, preprocessing, augmentation, and hyperparameter search budget. Compute is counted in total inference FLOPs per volume. We count the proposal as beating the baselines if gain routing matches full-3D mean Dice within 0.5 points at no more than 30% of full-3D FLOPs. It must also sit strictly above entropy routing on the Pareto curve at matched $\beta$ for the small organs: adrenal glands, duodenum, and pancreas.

Why not just…?

Why not just route by the 2D model's own uncertainty? This is the most serious threat. If uncertain regions and context-dependent regions largely coincide, the gain head adds nothing. Roth et al.'s low-fat pancreas failures suggest there is a class of regions that is highly uncertain but not fixable by context, and routing would waste compute on them. That is only suggestive. The matched-budget comparison between gain routing and entropy routing settles it directly. We will also report the rank correlation between $G$ and 2D entropy. If the correlation is high and the Pareto curves overlap, we will report that the simpler router suffices.

Why not just run an efficient 3D model everywhere? Networks like Swin-UMamba† shrink the cost of native 3D context considerably. If such a model matches 3D UX-Net accuracy at under 30% of its FLOPs, selective routing loses its purpose. We therefore include it on the same curve. Routing composes with it, since the efficient model can serve as the expensive arm. So the relevant test is whether routing still helps when that arm is cheap.

Why not just stack neighboring slices as input channels? Giving a 2D network $\pm k$ neighbors as extra channels (often called 2.5D) is cheap and often effective. It may be enough on its own, especially at thick slice spacing. We treat the slab model as a full baseline and also as an alternative expensive arm. If slab-everywhere matches 3D-everywhere, the remaining value of $G$ is diagnostic: it would tell us where depth matters, but it would not save compute.

Why not just smooth in 3D after the fact? Roth et al.'s filter is isotropic and blurs true boundaries along with the jagged ones. It is included as a baseline. We expect it to help large organs and hurt thin structures, and the per-organ breakdown will show whether that happens.

Proposed Experiments and Decisive Checks

The main experiment trains on AMOS CT and tests on AMOS CT, with FLARE 2021 as a second CT benchmark for replication. Metrics are per-organ Dice, a through-plane boundary-consistency score (the mean change in predicted organ contour between adjacent slices, relative to ground truth), and FLOPs per volume. We produce these on both native-spacing volumes and volumes resampled to 5 mm slices.

The first decisive check is the matched-budget comparison between gain routing and entropy routing on small organs. The second is whether the gap between the learned gain head and the oracle stays small; a large gap would mean $G$ is real but cannot be predicted from 2D input. The third is transfer to AMOS MRI from unseen scanners. If the head's ranking collapses there, the method is not robust to domain shift, and we will report that.

A negative result on the first check would still be informative. It would mean that a slice model's own uncertainty already tells you where depth context matters, which the field currently assumes without having tested it.