Spacing-Calibrated Weight Inflation for Low-Contrast 3D Grading

When a network pretrained on 2D images is reused on 3D scans such as CT or MRI, each flat filter has to be turned into a cube. Researchers call this weight inflation. There are two common ways to do it: copy the filter onto only the middle slice of the cube, or spread copies of it across neighbouring slices. Published papers disagree about which choice is better. We propose that both choices are points on a single scale, and that the right point is set by physics: how far apart the slices are in millimetres compared with the in-plane pixel size. Under this view, the depth profile of an inflated filter encodes an assumption about how much a structure blurs from one slice to the next. That assumption should be derived from the scan's voxel spacing, not picked by habit.

This matters most where 2D-to-3D transfer is most needed, which is small cohorts of anisotropic clinical scans ("anisotropic" meaning slices are much thicker than the in-plane pixel size). In those scans, faint lesion borders are smeared across thick slices. Fine-grained grading decisions, such as prostate cancer grade rather than cancer yes/no, depend on exactly those borders. With little training data, fine-tuning cannot undo a badly chosen starting point. We therefore expect the choice of inflation to matter most for boundary-band accuracy and for multi-grade labels, and to matter least for coarse binary tasks on large datasets.

Core Claim

Let a pretrained 2D filter $W$ be inflated into a 3D kernel whose depth slice $d$ holds $W \cdot g_\sigma(d)$. The profile is

$$ g_\sigma(d) = \frac{\exp\!\left(-(d\, s_z)^2 / 2\sigma^2\right)}{\sum_{d'} \exp\!\left(-(d'\, s_z)^2 / 2\sigma^2\right)}, $$

where $s_z$ is the slice spacing in millimetres and $\sigma$ is a physical width in millimetres. As $\sigma \to 0$ this reduces to centering inflation, which is the finding of Adapting Pre-trained Vision Transformers from 2D to 3D through Weight Inflation Improves Medical Image Segmentation. There, 2D BEiT weights placed on the central slice beat uniform averaging across 12 datasets. As $\sigma \to \infty$ the profile becomes average inflation. Intermediate values resemble the depth-Gaussian inflation that COVID Detection and Severity Prediction with 3D-ConvNeXt and Custom Pretrainings found best for ground-glass opacities.

Our falsifiable hypothesis concerns what stays fixed across datasets. We predict that the best $\sigma$, measured in millimetres, is roughly constant and tied to the filter's in-plane physical support, $\sigma^\star \approx c\, s_{xy} k$, where $s_{xy}$ is the in-plane pixel spacing, $k$ is the filter width in pixels, and $c$ is a constant. It follows that the best profile in slice units changes with slice thickness. Thick-slice data should favour centering. Near-isotropic data should favour spreading. If instead the best width in slice units is constant regardless of spacing, the hypothesis is wrong, and the literature's disagreement must come from backbone differences (ViT versus ConvNeXt) rather than geometry.

The file does not report the voxel spacing of the datasets used in those two papers. We therefore treat "spacing explains the contradiction" as a hypothesis to test, not a known fact.

Relation to Prior Work

Several papers motivate this proposal by exposing the gap. The BEiT inflation study and the 3D-ConvNeXt study reach opposite conclusions, and neither varies voxel spacing. The BEiT study also reports smaller gains on MRI than on CT and works with 5-slice windows. Both observations fit a geometric explanation, since clinical MRI is often acquired with thick slices. Bounding Maps for Universal Lesion Detection adds a related 2D observation: linearly decaying soft targets beat Gaussian ones for fuzzy lesions. This hints that the shape of a decay prior matters near ambiguous borders. We test linear profiles as a variant for that reason.

Other papers supply methods we reuse. From the BEiT inflation study we take the window-based ViT inflation recipe. The boundary-aware pooling loss of Promise: Prompt-driven 3D Medical Image Segmentation Using Pretrained Image Foundation Models serves as a fixed training loss. For small-sample classification, we attach the covariance-matrix head of A Geometric Multimodal Foundation Model Integrating Bp-MRI and Clinical Reports in Prostate Cancer Classification to the inflated backbone. That paper inflates BiomedCLIP's patch embedding for binary prostate cancer detection, but it does not study the inflation profile and does not attempt multi-grade labels.

Some prior work constrains the design. Reinventing 2D Convolutions for 3D Images sidesteps the profile question altogether with axial-coronal-sagittal (ACS) convolutions, which split channels across the three orthogonal planes. It is the main architectural competitor. 2D Convolutional Neural Networks for 3D Digital Breast Tomosynthesis Classification warns that slice-wise 2D models can beat native 3D models on small cohorts. Any gain we report must survive that comparison.

Our contribution differs from all of these in one respect. We change no architecture and add no loss. We change only the initialization rule, and we tie it to acquisition physics.

Research Program

The first part is a controlled geometry study. We take the cohorts with the finest native slice spacing and simulate thicker acquisitions. Averaging groups of slices mimics partial-volume blur and produces 1×, 2×, 3× and 5× slice thickness. Ground-truth masks are resampled the same way. The anatomy stays fixed, so the only thing that changes is spacing.

On each thickness we sweep $\sigma$ from centering to average inflation for two backbones: an inflated BEiT and an inflated ConvNeXt. We then check whether the optimal $\sigma$ lines up in millimetres or in slices. Training sizes are 10%, 25%, and 100% of the data, because the hypothesis predicts the effect should shrink as data grows.

The second part tests the rule on native clinical data. We apply the spacing-calibrated rule, which sets $\sigma$ from the fitted $c$ with no per-dataset tuning, to segmentation benchmarks with low-contrast targets and mixed spacings. These are BCV, Medical Segmentation Decathlon (MSD) Colon Tumor and Pancreas Tumor, and the Atrial Segmentation Challenge 2018 volumes. We report results inside a boundary band, meaning voxels within a fixed millimetre distance of the reference surface, as well as over whole volumes.

The third part asks whether the initialization effect grows with diagnostic granularity. On PI-CAI we fine-tune the same inflated backbone for binary clinically significant cancer detection and for 4-way ISUP grading, at matched small training sizes. We measure the gap between calibrated and fixed inflation at each label granularity. PROSTATE158 serves as the external test set.

Baselines

The naive reference point is a 3D network trained from random initialization, with the same architecture and training schedule. On BCV, Adapting Pre-trained Vision Transformers from 2D to 3D through Weight Inflation Improves Medical Image Segmentation reports this is 11.18% worse than pretrained centering inflation. We rerun it to anchor every learning curve.

The strongest direct competitor is fixed centering inflation of BEiT, the same paper's best configuration. It reaches 87.13 DSC on BCV, and matching or beating that number with our code is our first sanity check. It is the strongest competitor because it is already the published answer to our exact question.

The second competitor is fixed 2G Gaussian inflation from COVID Detection and Severity Prediction with 3D-ConvNeXt and Custom Pretrainings. That paper reports a severity cross-validation F1 of 0.6818 against 0.6271 from scratch, with multitask pretraining. We apply it with its published width.

Third, ACS convolutions from Reinventing 2D Convolutions for 3D Images, which report 86.76% average sensitivity on DeepLesion and 94.9% AUC on LIDC-IDRI. ACS avoids choosing a profile at all.

Fourth, a slice-wise 2D model with the same pretrained weights and averaged predictions.

For PI-CAI grading, the external reference is Causality-Driven One-Shot Learning for Prostate Cancer Grading from MRI. It reports a one-shot 4-way ISUP AUROC of 0.614 and 0.713 on a 2-way task under a Siemens-to-Philips scanner shift. For binary detection with 10% of the data, MFM-Geom reports an AUC-PR of 90.67. For ProMISe on MSD Colon Tumor, the file reports Dice 66.81 and Normalized Surface Dice (NSD) 81.24. However, ProMISe uses a point prompt, so we report it as a reference and do not treat it as a matched baseline. The file contains no nnU-Net result on these benchmarks, so we would run nnU-Net ourselves as a from-scratch 3D reference.

Two ablations isolate the key ingredient. The first uses the best $\sigma$ found by sweeping in slice units, with no physical calibration. This tells us whether millimetres are the right units. The second makes $\sigma$ a learnable parameter per layer, initialized at centering. This tells us whether the prior has to be set in advance or whether training can find it.

For fairness, every method uses identical data splits, augmentation, loss, optimizer, epochs, and an equal hyperparameter search budget. The inflation width counts as one hyperparameter for every inflation baseline. The calibrated rule gets none, because $c$ is fitted only on the synthetic study.

"Beating" a baseline means the calibrated rule matches the best per-dataset-tuned fixed profile, within seed variance over five seeds, while using zero per-dataset tuning. It must also beat centering inflation on boundary-band Dice or NSD in the 10% and 25% data regimes on at least the thick-slice datasets.

Why not just…?

Why not just resample every volume to isotropic spacing and use one fixed inflation? This is the most serious threat. If it works equally well, our rule reduces to "resample first." Resampling thick slices invents interpolated boundary content, and it multiplies depth by the anisotropy ratio. That runs into the memory limits which already force 5-slice windows in the BEiT study. We include isotropic resampling combined with the best fixed profile as a baseline, at matched GPU memory. If it matches calibrated inflation at equal compute, we will report that resampling suffices.

Why not just let fine-tuning learn the depth weights? With enough data, it will. The hypothesis itself predicts that calibrated and fixed profiles converge at 100% of the data. The learnable-$\sigma$ ablation and the data-size sweep test directly whether the prior matters only when data is scarce. If learnable $\sigma$ started at centering matches calibration at 10% data, the physics-based rule adds nothing, and we will say so.

Why not just use ACS convolutions or slice-wise 2D models and avoid the question? ACS keeps the 2D parameter count and reports strong LIDC-IDRI and DeepLesion results. Slice-wise models are hard to beat on small cohorts, as the tomosynthesis study shows. Both are run under matched conditions. Our claim is narrower: among inflation methods, a physically set profile removes a tuning step and helps most at borders on anisotropic data. If ACS beats every inflation variant in the boundary band, inflation is the wrong lever, and that would be a useful negative result.

Why not just add a boundary loss instead of changing initialization? Boundary losses clearly help. In Combating Uncertainty with Novel Losses for Automatic Left Atrium Segmentation, the Overlap Loss raised Dice from 82.01% to 89.83%. But losses and initialization are complementary rather than rival choices. We keep the ProMISe boundary loss fixed across all runs. In a secondary run we also add the Overlap Loss, to check that the initialization gain does not disappear once a strong boundary loss is present.

Proposed Experiments and Decisive Checks

The first decisive check is the synthetic thickness study. We plot the optimal $\sigma$ against simulated slice thickness in two units. If the optimum stays flat in millimetres and shifts in slice units, the hypothesis holds. If it stays flat in slice units, the hypothesis fails. We also check whether the answer depends on backbone, because a ViT-versus-ConvNeXt split would be the alternative explanation for the published disagreement.

The second check is transfer without tuning. With $c$ frozen from the synthetic study, calibrated inflation must match the per-dataset sweep's best fixed profile on BCV, MSD Colon and Pancreas Tumor, and the Atrial data. We report whole-volume Dice, NSD, and boundary-band Dice at 10%, 25%, and 100% of the training data.

The third check addresses granularity. On PI-CAI, the gain from calibrated over centering inflation should be larger for 4-way ISUP AUROC than for binary AUC-PR at the same training size, and should persist on PROSTATE158. If the gain is equal or smaller for fine-grained grading, we drop the link between granularity and boundaries and keep only the segmentation claim.