Hybrid networks for 3D medical images use a convolutional network for local detail and a transformer for long-range context. They pay for that context with coarse inputs. Self-attention compares every token with every other token, so models such as UNETR cut a CT volume into 16×16×16-voxel patches before the transformer sees it. Fine boundaries are lost at the start and must be recovered by the decoder. Making patches four times smaller in each direction multiplies the token count by 64 and the attention cost by about 4,000, which is out of reach on one GPU.
We propose to keep fine tokens and estimate their global attention by random sampling, using the coarse attention the model already computes as a control variate. A control variate is a cheap quantity that is correlated with the noisy estimate and whose exact value is known. Subtracting its sampled version and adding back its exact version cancels most of the noise. Concretely, each query attends exactly to coarse summaries of every region of the volume. It then samples only a few regions and corrects the coarse answer using the fine tokens inside them. The estimate stays unbiased, and its variance depends only on how much fine tokens differ from their region's summary. Convolutional features make that difference small in smooth anatomy and large mainly near boundaries, which is where the extra resolution is needed.
Core Claim
Define a two-level attention operator over a 3D grid of fine tokens grouped into coarse cells of $m$ fine tokens each. For a query $q$, let $P_c$ be the softmax probability of cell $c$ computed against pooled cell keys. Let $\bar v_c$ be the pooled cell value. Let $y_c$ be the exact softmax attention output of $q$ over the $m$ fine tokens inside cell $c$. The target output is $y = \sum_c P_c\, y_c$, a well-defined hierarchical attention and not an approximation of something else. Computing it exactly costs as much as full attention. We instead draw $S$ cells $c_1,\dots,c_S$ from a fixed copy $Q$ of $P$ (fixed meaning no gradient flows through it) and form
$$\hat y = \sum_c P_c\, \bar v_c \;+\; \frac{1}{S}\sum_{s=1}^{S} \frac{P_{c_s}}{Q_{c_s}}\big(y_{c_s} - \bar v_{c_s}\big).$$
The first term is exact and cheap. The second term is an importance-weighted correction with expectation $\sum_c P_c(y_c - \bar v_c)$, so $\mathbb{E}[\hat y] = y$. Because $Q$ is held fixed, gradients flow pathwise through $P_c$, $y_c$ and $\bar v_c$. That makes the gradient of $\hat y$ an unbiased estimate of the gradient of $y$ without a score-function term. The variance is $\tfrac{1}{S}\mathrm{Var}_{c\sim P}(y_c - \bar v_c)$, compared with $\tfrac1S\mathrm{Var}_{c\sim P}(y_c)$ without the control variate. The method helps exactly when fine values cluster around their cell means. The query's own cell and its immediate neighbours are always computed exactly, so the local receptive field is never sampled.
For a 96³ crop with 4³-voxel tokens, there are 13,824 fine tokens and 216 cells of 64 tokens. The cells match UNETR's original patch size. Per head, exact coarse attention plus $S=4$ sampled cells costs on the order of seven million query-key products, against roughly 190 million for full fine attention. The claim is that, at matched memory, this operator gives better boundary accuracy (HD95, the 95th-percentile surface distance between predicted and true boundaries) than coarse-patch hybrids and than token pruning, while keeping region-overlap (Dice) accuracy.
Relation to Prior Work
Three papers motivate the gap. UNETR runs a vision transformer encoder on 16³ patches with a convolutional decoder. TransBTS and ViT-V-Net put a transformer bottleneck after heavy convolutional downsampling. All three accept coarse tokens to keep dense attention affordable, which is the resolution limit we target. We keep their encoder–transformer–decoder layout and change only how the transformer reaches fine tokens.
Two papers supply the method. Stochastic Sparse Attention for Memory-Bound Inference builds unbiased sampled attention for language-model decoding and reduces its variance with stratified and systematic sampling. It still computes all query-key scores, works only at inference, and handles 1D sequences. Our estimator avoids the full score matrix through the coarse level, is used during training, and reduces variance with a control variate rather than by stratifying. A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation pairs cheap low-fidelity simulations with scarce high-fidelity rollouts. Our coarse attention is the low-fidelity model and the sampled fine cells are the high-fidelity rollouts.
The general principle of a cheap correlated proxy with known mean comes from two papers. Variance Reduction for Stochastic Gradient Optimization uses moment-based Taylor proxies with reported correlation above 0.96. Reducing Reparameterization Gradient Variance reports 20x to 2,000x variance reduction from linearized proxies. These papers also warn us. Their proxies weaken where the function is sharply nonlinear, and here that means at organ boundaries. We measure the variance ratio directly instead of assuming it.
The closest competitor for the same compute budget is Token Sparsification for Faster Medical Image Segmentation. It prunes UNETR tokens with a Straight-Through Gumbel Soft-topK selector and reports a slight HD95 increase. Pruning removes tokens for good using a gradient-biased selector. Our method keeps every region's coarse contribution and samples fine corrections without bias.
The non-transformer routes are CCNet, which uses row and column attention, and DAS, which uses deformable sampling. Both give global reach deterministically but along fixed or learned geometric paths, not content-weighted over the whole volume. We do not use discrete-latent estimators such as RELAX or RODEO. Holding the sampling distribution fixed makes the gradient pathwise, so a learned surrogate is unnecessary.
Research Program
We will implement the operator as a drop-in attention layer in the UNETR pipeline from the TokenSparse-for-MedSeg codebase. The data handling and decoder stay unchanged. A shallow convolutional stem produces 4³ tokens. A strided convolution produces cell tokens, and cell keys and values are pooled from them.
The analysis has three parts. First, we log the per-layer variance of $\hat y$ and of the attention-parameter gradients over training, with and without the control variate. We break this down by whether the query lies near a ground-truth boundary. Second, we sweep $S \in \{1,2,4,8\}$ and the cell size to trace accuracy against memory. Third, we test whether an optional penalty on within-cell value spread improves the variance ratio without hurting Dice. This is the one place where the convolutional inductive bias could be shaped on purpose.
Baselines
The naive reference point is UNETR at 16³ patches, retrained in our pipeline. Its leaderboard result is an average Dice of 0.891 on BTCV (UNETR). That figure comes from the free-competition setting, so we report our own retrained number under identical crops and schedules rather than claiming to beat 0.891 directly. The coarse-only ablation sets $S=0$ and keeps only the exact cell-level term. It isolates whether sampled fine corrections add anything beyond a hierarchical coarse model.
The strongest existing efficiency method is Straight-Through Soft-topK token pruning (Token Sparsification). On MSD BraTS it reports MACs cut from 824G to 428G and a 120% training-throughput gain, with Dice "equal to or slightly higher than" UNETR. The file gives no numeric Dice or HD95 for this method. We run its released code with its default settings and also apply it to the 4³-token model at matched peak memory. Beating it means equal Dice with significantly lower HD95 at equal memory, measured over three seeds.
For the deterministic structured alternative, we replace our operator with 3D criss-cross attention in the style of CCNet at the same token resolution. For the key ingredient, two ablations matter most. The first samples cells exactly as ours does but without the control variate, which is plain importance sampling. The second applies stratified sampling over cells in the S²ANTA style without the control variate. Where memory allows, which means smaller crops, full fine attention serves as an upper bound. All methods share data splits, augmentation, optimizer, epochs, and a fixed tuning budget of eight learning-rate and weight-decay trials each.
Why not just…?
Why not just pick the top-S cells deterministically? Exact attention over the S highest-probability cells has zero variance and is simpler. This is a genuine threat. It is biased, though, because it always ignores mass outside the top cells. Its gradient never reaches cells it skips, so cell attention may lock in early. We run it at identical S. If it matches our method on Dice and HD95, the sampling adds nothing and we will say so.
Why not just use the coarse hierarchy alone, or plain 16³ patches? That is the $S=0$ ablation. If fine corrections do not improve HD95 over it, the premise that fine tokens matter for global context fails. We first check this premise cheaply with full fine attention on small crops.
Why not just prune tokens with a straight-through selector? Token Sparsification already gets large speedups at nearly unchanged Dice. Its own reported HD95 increase is the opening we target, and the matched-memory comparison above settles the question.
Why not just use criss-cross or deformable attention? These are deterministic and hardware-friendly, but their routing follows fixed axes or local offsets. The CCNet-style baseline at equal resolution tests whether content-weighted global sampling matters for 3D anatomy.
Proposed Experiments
The main benchmarks are BTCV, where small-organ Dice and HD95 are reported separately for the adrenal glands and gallbladder, and MSD BraTS. We add BraTS 2019 against the TransBTS figures of 78.93% enhancing tumor, 90.00% whole tumor and 81.94% tumor core Dice (TransBTS). We report Dice, HD95, peak memory, training and inference throughput, and measured attention-output and gradient variance.
Three checks decide the idea. First, the control variate must cut sampled-term variance by at least 4x on trained models, including near boundaries. If the ratio is below 2x, the mechanism is not doing the work. Second, at matched memory, our model must beat both straight-through pruning and the top-S deterministic variant on HD95 without losing Dice. Third, on small crops where full fine attention fits, our model should come within seed noise of it. That would show the estimator's noise is not what limits accuracy.
References
- UNETR: Transformers for 3D Medical Image Segmentation
- TransBTS: Multimodal Brain Tumor Segmentation Using Transformer
- ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration
- Stochastic Sparse Attention for Memory-Bound Inference
- A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation
- Variance Reduction for Stochastic Gradient Optimization
- Reducing Reparameterization Gradient Variance
- Token Sparsification for Faster Medical Image Segmentation
- CCNet: Criss-Cross Attention for Semantic Segmentation
- DAS: A Deformable Attention to Capture Salient Information in CNNs
- Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
- Gradient Estimation with Discrete Stein Operators