Transformer segmentation models for 3D medical scans, such as Swin UNETR, keep computation affordable by computing self-attention only inside small fixed boxes of image patches (windows). Every other layer, they slide the box grid by half a window so that information can cross box borders. This is shifted window self-attention. The grid is placed at a fixed position relative to the image crop, with no regard for anatomy. For a liver this does not matter. For a 4 mm kidney tumor, an intercostal artery, or an enhancing tumor rim that spans only a few patches, it may matter a great deal. Whether the structure lands inside one window or is cut across two or four windows is an accident of where the crop began. When it is cut, each fragment is averaged together with a window full of background patches. Medical segmentation is dominated by extreme class imbalance: background often covers more than 99% of voxels. The structures most likely to be cut are therefore exactly the rare classes that clinicians care about.
This note proposes two things. The first is to measure this effect directly. We treat the grid offset, which we call the "phase," as a hidden variable and ask how much rare-structure accuracy changes when only the phase changes. The second is a cheap fix, conditional on the effect being real: at inference, a small helper network guesses where small foreground structures are, and the model picks, separately for each input crop, the window-grid offset that keeps those structures whole inside single windows. This needs no new attention kernel. The only change is the amount of cyclic shift applied before the windows are formed.
Core Claim
We make two claims. First, in shifted window segmentation models, accuracy on small minority structures depends measurably on grid phase, and the dependence grows as structure size falls below the window's physical footprint. Large organs should be nearly phase-invariant. Second, choosing the phase per crop to keep predicted small structures inside single windows recovers most of that lost accuracy in about 1.1 forward passes. Averaging predictions over many phases would recover it at K times the cost. The gains should add to those from loss-level imbalance correction, because the two act on different parts of the model: one on what attention sees, the other on which output pixels get gradient.
Relation to Prior Work
Several papers motivate the gap. Swin UNETR puts a 3D shifted window encoder in front of a convolutional decoder for brain tumors. It keeps window sizes and shift trajectories fixed regardless of lesion scale, and it never asks whether small sub-regions are split. CIS-UNet places shifted window attention only at the bottleneck to segment the aorta and 13 thin, imbalanced branches from CT angiography. It still reports broken branch continuity, for example in the left renal artery, and confusion around the celiac artery. Its attention is uniform over space, and it is trained with plain Dice plus cross-entropy. Neither paper varies where the grid falls. We take their failures on thin structures as the motivating symptom and test whether grid phase is one cause.
Two papers suggest designs we deliberately avoid. ClassFormer argues that fixed window partitioning fragments boundaries between organs. It replaces the window grid with class-aware saliency sampling and class prototype tokens, and reaches 84.40% Dice on Synapse. It works on 2D slices and adds substantial decoder machinery. We keep the standard shifted window backbone and change only one integer per crop, which is far easier to adopt in existing 3D pipelines. The survey Transformers in computational visual media notes that masked, shifted attention is already awkward for hardware. That is why we rule out learned deformable or irregular windows, and it matches the concern behind the re-indexing approach of Flash3D.
A Unified Loss for Handling Inter-Class and Intra-Class Imbalance supplies an orthogonal tool that we reuse. Its Balance Loss mines hard background pixels to match the foreground count and down-weights easy interior pixels. It lifts KiTS19 kidney-tumor Dice from 59.8 to 66.8 over cross-entropy, but it acts only at the network output. We use it as a strong baseline and as a component we combine with.
Research Program
The work starts with a phase audit. Shifting the window grid by an offset $o$ is exactly equivalent to rolling the token map by $o$ before window partitioning and rolling it back afterwards, at both the regular and the shifted layers. We implement this as one extra cyclic roll with the existing attention mask logic, so trained models can be evaluated at any phase without retraining.
Offsets are whole patches, from 0 to $M-1$ along each axis, where $M$ is the window size in patches. This isolates the window grid from sub-patch effects of the patch embedding. As a control, we separately translate the input by sub-patch amounts. Offsets are applied only in the first two stages, for two reasons. After patch merging, an offset of one stage-1 patch no longer maps to a whole number of tokens. Also, deeper-stage windows cover large physical regions, where fragmenting tiny structures is not the relevant failure.
For each test case and structure, we record Dice across all offsets, i.e. every combination of per-axis offsets. The phase sensitivity is the spread across offsets:
$$S_c = \mathbb{E}_{\text{cases}}\big[\max_o \mathrm{Dice}_c(o) - \min_o \mathrm{Dice}_c(o)\big],$$
where $c$ indexes a structure class, $o$ ranges over the audited offsets, and $\mathrm{Dice}_c(o)$ is that class's Dice at offset $o$. We stratify $S_c$ by the ratio of a structure's extent to the window's physical footprint. The first claim predicts that $S_c$ is near zero for large organs and rises sharply once this ratio falls to about one or below.
The fix, phase anchoring, then works as follows. A small 3D convolutional locator with a few hundred thousand parameters predicts a coarse foreground map for the minority classes only. For each crop, we score each candidate offset by how completely the locator's predicted connected components sit inside a single window, under both the regular and the shifted partition. We run the main model at the best-scoring offset.
During training, we anchor the offset on the ground-truth mask with probability $p$ and otherwise sample it uniformly. The model therefore sees both anchored and arbitrary phases, and the train and test phase distributions match. An oracle variant anchors on ground truth at test time. This gives an upper bound that tells us how much of any shortfall comes from the locator rather than from the anchoring idea itself.
Baselines
The naive reference is the unmodified shifted window model at its default phase, trained with Dice plus cross-entropy. On BraTS 2021, Swin UNETR reports 5-fold cross-validation Dice of 0.891 for enhancing tumor, 0.933 for whole tumor and 0.917 for tumor core Swin UNETR. On the Aorta CTA dataset, CIS-UNet reports a mean Dice of 0.713 and a mean surface distance of 2.78 mm, against 0.697 and 3.39 mm for SwinUNetR CIS-UNet. CIS-UNet is the strongest existing shifted window method on the thin-branch problem that most directly tests our claim. We rerun both models under our splits rather than quoting these numbers as the comparison. On Aorta CTA, beating CIS-UNet means a gain in mean Dice over our own CIS-UNet rerun, concentrated in the small branches, with fewer disconnected branch components.
The strongest loss-level competitor is the same backbone trained with Balance Loss. We also include focal loss, each with a comparable tuning budget. For KiTS19, the only reported Balance Loss figure is 66.8 tumor Dice, from its own CNN setting A Unified Loss. We found no reported result on Lacuna for Balance Loss on a shifted window backbone.
The strongest architecture-agnostic reference is a self-configuring convolutional 3D U-Net of the kind that won KiTS19. That model reported kidney Dice of 0.974 and tumor Dice of 0.851 on the challenge test set KiTS19 challenge. We retrain it on our splits rather than comparing against the challenge figure.
Two further baselines isolate the key ingredient. Random-phase training augmentation with a single default-phase pass at test time uses the same training compute but no anchoring. Shift test-time averaging over all offsets is the brute-force way to remove phase dependence.
Every method gets the same data splits, the same crop size and sliding-window inference, the same number of training epochs, and the same count of hyperparameter trials. We report inference cost in forward passes. Anchoring counts as a win only if two conditions hold. It must beat random-phase augmentation on small-structure Dice at matched cost. It must also reach at least most of shift test-time averaging's gain at roughly one-eighth or less of its inference cost.
Why not just…?
Why not just use nnU-Net and skip transformers? On KiTS19 a plain convolutional U-Net beat 105 other teams KiTS19 challenge, so this is a real threat. We do not claim anchoring will beat it everywhere. On BraTS, Swin UNETR outperformed nnU-Net in cross-validation Swin UNETR, so shifted window models are worth fixing where they are already competitive. The phase audit also tests a concrete hypothesis about why they lose on tiny structures. Convolutions are close to translation-equivariant, so a convolutional model should show near-zero $S_c$, while the shifted window model should not. If anchoring closes the small-structure gap to the U-Net, that is the finding. If the gap remains, phase is not the main explanation, and we report that.
Why not just apply Balance Loss or focal loss to a standard Swin backbone? Loss reweighting changes which output voxels receive gradient. It cannot change the fact that a lesion's patches were split across windows and averaged with background before any prediction is made. The paper that introduced Balance Loss also reports that weighted cross-entropy, focal and Tversky losses need dataset-specific tuning A Unified Loss. The decisive test is the 2×2 comparison of anchoring against loss choice. If Balance Loss alone drives $S_c$ to near zero, anchoring is redundant and we will say so.
Why not just train with random shifts, or average predictions over shifts at test time? Random-phase training could make the model robust to fragmentation. If it drives $S_c$ to near zero, the problem is solved more cheaply and anchoring is unnecessary; the audit on randomly augmented models settles this directly. Test-time averaging almost certainly removes phase variance, but it costs up to $M^3$ passes over every crop of a 3D volume. Its value here is as the ceiling that anchoring must approach cheaply. Smaller windows are a third option. They reduce fragmentation of tiny structures but shrink the context available to large ones, so we include a window-size sweep for comparison.
Proposed Experiments
We audit and intervene on four benchmarks that span the size range. BraTS 2021 has small enhancing rims inside larger tumors. KiTS19 has tumors inside kidneys. The Aorta CTA dataset, with 44 training and 15 test scans, has 13 thin branches. Synapse multi-organ CT serves as a large-organ control, since adrenal glands and pancreas sit next to liver and spleen.
The metrics are per-class Dice, mean surface distance, and phase sensitivity $S_c$ stratified by structure size relative to the window. For vessels, we also count connected components per branch as a measure of continuity.
The first decisive check is the audit on the default and random-phase-augmented Swin UNETR and CIS-UNet. If $S_c$ for structures smaller than a window is not clearly larger than for large organs and larger than the convolutional control, the premise fails, and the project ends as a negative result. The second check compares anchored against oracle-anchored against random-phase models on Aorta CTA branches and KiTS19 tumors. It tells us whether single-window containment, rather than extra phase diversity, is what helps. The third check stacks anchoring with Balance Loss to test whether the two gains add. The fourth reports cost against accuracy for anchoring, test-time averaging and the U-Net, so that clinical users can see the trade-off directly.
References
- Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images
- CIS-UNet: Multi-class segmentation of the aorta in computed tomography angiography via context-aware shifted window self-attention
- ClassFormer: Exploring Class-Aware Dependency with Transformer for Medical Image Segmentation
- Transformers in computational visual media: A survey
- Flash3D: Super-scaling Point Transformers through Joint Hardware-Geometry Locality
- A Unified Loss for Handling Inter-Class and Intra-Class Imbalance in Medical Image Segmentation
- The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge