Spending Transformer Tokens Where the Anatomy Is Complex

Hybrid networks for 3D medical scans use convolutions for local detail and a transformer (a model in which every token can attend to every other token) for long-range context. They all pay the same price: attention cost grows with the square of the number of tokens, so the volume is cut into a fixed coarse grid before the transformer ever sees it. A homogeneous stretch of liver and a tangle of vessels, pancreas and bowel receive the same number of tokens. We propose to allocate the transformer's token budget unevenly across the scan. The allocation is driven by the local intrinsic dimension of the network's own features, which is the number of independent directions along which neighboring feature vectors actually vary. This quantity can be measured from unlabeled scans after self-supervised pretraining. The allocation is therefore fixed before any expert label is used, which matters when only a handful of annotated volumes exist.

In two sentences: after pretraining a hybrid CNN–transformer on unlabeled scans, we measure how many dimensions the features vary along in each region, and we give fine-grained transformer tokens to high-dimensional regions and a single merged token to low-dimensional ones, at a fixed total token count. Boundaries are already handled at full resolution by the convolutional skip path, so the scarce global tokens should go where the context is complex, not merely where edges are.

Core Claim

At a matched token and compute budget, intrinsic-dimension-guided token allocation in a hybrid segmentation network improves Dice and 95% Hausdorff distance over uniform tokenization and over simpler complexity signals. The gain should be largest when labels are scarce (1–10%). The signal is label-free, so it adds no parameters that must be fit from the few labeled scans. The claim is falsifiable in a specific way. If allocating by plain feature variance (how much features vary) does as well as allocating by intrinsic dimension (how many directions they vary along), then the geometric story adds nothing.

The distinction is real at organ boundaries. There, features change sharply but along essentially one direction (across the boundary), so variance is high while intrinsic dimension is about 1. Variance-based allocation would spend tokens on every edge. Intrinsic-dimension-based allocation leaves edges to the convolutional decoder and spends tokens on textured tumors and crowded multi-organ junctions.

Grounding in Prior Work

Theory motivates the allocation rule. Chen et al. prove that deep ReLU networks can approximate functions on a $d$-dimensional manifold with a size exponent governed by $d$ rather than the ambient dimension. They build a chart-finding sub-network followed by local Taylor fits. Chui, Lin and Zhou build localized networks whose rate $\mathcal{O}(m^{-2s/(2s+d)})$ depends only on intrinsic dimension. Liu et al. and Shen et al. extend this adaptation to convolutional residual networks and CNN classifiers. All four give existence results for a single global $d$. We use their conclusion heuristically: approximation difficulty is set by local dimension, so resolution should follow it. We then test this empirically rather than proving it for attention.

Two empirical papers supply the method template, and we move it to a new axis. Geometry-Guided Layerwise FFN Width Allocation measures per-layer intrinsic dimension with TwoNN (an estimator that uses only the ratio of each point's second- to first-nearest-neighbor distance). It allocates feed-forward width in language models through the surrogate $\text{Error}\propto \text{Width}^{-2s/d_{\text{int}}}$. It reports a 0.019 validation-loss gain at 440M parameters, 6.3× the gain of a cosine taper, but it needs a separately trained reference model. Exploring Intrinsic Dimension for VLM Pruning shows that high-dimensional vision layers collapse under pruning while low-dimensional language layers do not. Weighting importance by dimension lifts CIDEr at 90% pruning from 93.1 to 106.4. That result supports our premise that high-dimensional representations are where compression hurts. Both works allocate across layers or weights. We allocate across space, per scan. We also use the self-supervised pretraining run itself as the reference model, which removes the extra training run that the width-allocation method requires.

IdEst seems at first to cut against us: it finds that lower intrinsic dimension of a representation predicts better linear-probe accuracy. The quantities differ, though. IdEst measures the global dimension of image embeddings across a dataset, where low values mean well-organized classes. We measure the dimension of feature vectors at neighboring positions within one scan, which describes spatial complexity. IdEst does lend support to one step we rely on: intrinsic dimension of self-supervised features carries useful signal without labels.

The architectural baselines constrain the design. TransBTS and UNETR both feed a transformer from a fixed-stride grid. UNETR is stuck at 16³ patches for memory reasons and notes that 2D ImageNet pretraining does not help. Light-UNETR with CSE cuts FLOPs by up to 90.8% through uniformly downsampled attention plus convolutional branches, and pairs this with semi-supervised consistency. Its downsampling is identical everywhere, and that is the design choice we change. UniMiSS shows that domain-native self-supervision makes 40% labels beat 100% on BTCV. This motivates doing the pretraining in-domain before measuring geometry.

Research Program

The network has three parts. A 3D convolutional stem downsamples by 4 and feeds full-resolution skip connections to a convolutional decoder, as in TransBTS. The volume is partitioned into windows of 16³ voxels, which corresponds to 4³ stem-feature cells. Each window is represented by 1, 8 or 64 tokens: its cells are pooled to 1³, 2³ or kept at 4³. Every token carries a positional encoding of both location and scale. A standard transformer attends over this mixed-resolution set, and its outputs are scattered back onto the grid for the decoder.

Pretraining uses masked volume reconstruction on unlabeled scans with uniform 8-tokens-per-window tokenization. After pretraining, for each window $w$ we estimate its local intrinsic dimension $d_w$ by applying TwoNN to its 64 stem features together with those of adjacent windows. This is cheap: roughly a few hundred points per window, with nearest-neighbor search only.

The allocation follows the surrogate from the FFN-width paper. We choose token counts $n_w$ to minimize the total predicted error subject to a budget $N$:

$$\min_{\{n_w\}} \sum_w n_w^{-2s/d_w} \quad \text{subject to} \quad \sum_w n_w = N,$$

where $s$ is a smoothness constant tuned once on unlabeled reconstruction loss. Each $n_w$ is then rounded to the nearest of {1, 8, 64}. This allocation gives more tokens to windows with larger $d_w$. To avoid computing TwoNN at test time, a two-layer convolutional head is trained on unlabeled scans to regress the map of $d_w$ values. This head uses no labels.

Fine-tuning then uses 1–10% of labeled scans, with the allocation frozen. The analysis side checks whether the $d_w$ maps are stable across scans and pretraining seeds. It also checks whether they differ spatially from feature-variance maps, concentrating on textured lesions and organ junctions rather than tracing every edge. Finally, it checks whether $d_w$ predicts where a uniform-token model makes its per-window errors.

Baselines

The naive reference is the same hybrid with the same pretraining and uniform tokenization: 8 tokens per window everywhere, the TransBTS-style fixed grid. It is compared at an identical total token count, identical FLOPs, identical fine-tuning schedule and identical hyperparameter search budget. Beating it means higher mean Dice at equal FLOPs, or equal Dice at noticeably lower FLOPs, across at least three seeds with non-overlapping intervals.

The strongest existing method for the label-scarce setting is CSE-Light-UNETR from Light-UNETR. It is strongest because it already combines a lightweight hybrid attention block with semi-supervised training, and it is the only method in the file reporting both a FLOPs reduction and a label-efficient result on the same benchmark. On LA with 10% labels it is reported to exceed BCP by 1.43% Jaccard. The file gives no absolute Jaccard value, so no reported result was found on Lacuna for CSE-Light-UNETR's absolute LA score, and we will reproduce it from its description. On fully supervised BraTS 2019, Light-UNETR-L reports 84.32% Dice at 59.45G FLOPs. For fairness, our model gets the same unlabeled data, either as our pretraining or as its semi-supervised pool. We also run a variant that adds CSE-style consistency to our model, so that the tokenization change is separated from the training recipe. Beating it means matching 84.32% BraTS Dice below 59.45G FLOPs, and exceeding the reproduced CSE-Light-UNETR Jaccard on LA at 10% labels.

For BTCV, the reference points are UNETR's reported 0.891 average Dice with full labels and UniMiSS's reported 80.97% Dice at 40% labels. UniMiSS is used as the label-scarce anchor.

The ablations that isolate the key ingredient all keep the budget $N$ fixed and change only the allocation signal. They are random allocation; raw voxel intensity variance; stem-feature variance, which is the critical control; a learned token scorer trained end-to-end on the labeled fraction; intrinsic dimension computed from a randomly initialized, unpretrained stem; and inverted allocation, which gives low-dimension windows more tokens. The claim survives only if intrinsic dimension beats feature variance and random allocation, and if inverted allocation does worse than uniform.

Why not just…?

Why not just use linear-cost attention or a state-space model and keep every token? Linear-time models such as EccoMamba remove the quadratic bottleneck. They still spend equal computation per location, and they need multi-directional scanning to avoid flattening artifacts. That leaves open the question of whether compute should be uneven. This alternative is a genuine threat. We therefore include a linear-attention hybrid at matched FLOPs using the full fine grid. If it matches our allocation, the conclusion is that uniform fine tokens at linear cost beat selective tokens at quadratic cost. That would be a useful negative result.

Why not just learn which tokens to keep, as dynamic token pruning does? A learned scorer must be fit from the labeled scans, and in the 10% LA regime these number only a few volumes. That is exactly the setting in which Heuillet et al. find that extra flexible capacity generalizes worse than strong fixed priors. Our ablation trains such a scorer on the same labels. We expect it to match us at 100% labels and fall behind at 1–10%. If it does not fall behind, the label-free argument fails.

Why not just refine tokens where the image is busy, using intensity or feature variance? This is the most plausible competitor, and it is the control that decides the project. Variance and intrinsic dimension disagree mainly at boundaries, where variance is high but dimension is low. Our argument is that the convolutional skip path already resolves boundaries. The stem-feature-variance ablation tests this directly. If it matches intrinsic dimension on all three datasets, we will report that the simpler signal is sufficient.

Why not just use Light-UNETR's uniform downsampled attention? It is our main baseline. Its reported efficiency comes from shrinking attention everywhere, while our change is orthogonal: it decides where to shrink. The two can be combined, and we test that combination.

Proposed Experiments

We pretrain on the unlabeled scans of each benchmark and then fine-tune at 1%, 10%, 40% and 100% labels. On BTCV multi-organ CT we report per-organ Dice, with small organs such as the gallbladder and adrenal glands singled out. On BraTS 2019 we report enhancing tumor, whole tumor and tumor core Dice together with 95% Hausdorff distance, against TransBTS's reported 78.93%, 90.00% and 81.94% and Light-UNETR-L's 84.32%. On LA we report Dice and Jaccard at 10% and 5% labels.

The main figure is a Dice-versus-FLOPs curve at four token budgets for each allocation signal. We also verify the mechanism directly by measuring the correlation between $d_w$ and the per-window error of the uniform model. It should be positive and higher than the correlation for feature variance. We further check that windows with high $d_w$ fall on lesions and organ junctions rather than on smooth edges.

The decisive checks are three. First, intrinsic dimension must beat stem-feature variance and random allocation at matched budget on at least two of the three datasets. Second, the advantage must grow as the label fraction shrinks. Third, dimension maps from an unpretrained stem must give no gain, showing that the signal comes from self-supervised geometry and not from raw image statistics. If the first check fails, we abandon the geometric claim and report the variance baseline as the practical result.