Attention over a 3D scan is expensive because every voxel patch attends to every other. A volume with side length $L$ yields about $L^3$ tokens, so full attention costs about $L^6$, and doubling resolution costs 64 times more. The usual fixes shrink the problem by rule. They downsample, cut the volume into windows, keep only non-empty voxels, or keep only high-variance patches. None of these rules asks whether the tokens that survive mean anything.
Separately, work on identifiability asks when a network's internal variables are guaranteed to match the real factors that generated the data, rather than some arbitrary scrambling of them. One recent result, Sparling, shows that if a network is forced through an extremely sparse, strictly local intermediate layer (roughly 99.9% of activations zeroed), then the few activations that survive provably correspond to the true local "motifs" in the data. This holds when the network is trained only on the end task, with no concept labels.
This note proposes to use that one sparse layer for two jobs at once. It picks which few locations in a 3D volume become attention tokens, and it guarantees that those tokens are identifiable anatomical or structural motifs. In plain terms, a cheap local network scans the whole volume. A hard top-quantile gate keeps only a tiny fraction of locations. Global attention then runs only over the survivors. The sparsity that makes attention affordable is the same sparsity that makes the tokens trustworthy.
Core Claim
Let $N \approx L^3$ be the number of voxel locations, and let $\delta$ be the fraction a hard quantile gate keeps. A shallow local 3D convolutional encoder costs $O(N)$. Attention over the $K=\delta N$ surviving tokens costs $O(\delta^2 N^2)$.
At $256^3$ ($N \approx 1.7\times 10^7$) with $\delta = 10^{-4}$, that leaves about 1,700 tokens. Dense attention over that many tokens is routine.
The claim has two parts. First, Sparling's Motif Identifiability Theorem carries over to this 3D architecture when the model is trained end-to-end on a volume-level or segmentation objective. In that setting the gated activations recover ground-truth motif locations and types up to permutation. Second, the identifiability comes from the extreme, annealed sparsity and not from the token budget alone. Other token selectors given the same budget $K$, such as variance maps, farthest-point sampling, or moderately sparse learned gates, reach similar compute but do not recover the motifs.
If the second part fails, the identifiability framing adds nothing over existing token pruning, and the proposal should be dropped.
Where This Sits
Sparling (Sparling) supplies the method and the theorem. It introduces the quantile-threshold sparsity layer and proves motif identifiability. It was only tested on 1D and 2D data (DigitCircle, LaTeX-OCR, AudioMNISTSequence, RNA splicing), with dense convolutional downstream networks and no attention. It also relies on motifs not overlapping in space. The present work moves it to 3D grids, puts global attention downstream, and measures directly how overlap breaks it.
Three volumetric methods already prune tokens by rule, and they motivate the efficiency side:
- Omni-fMRI drops background patches using a variance map, going from about 14,000 to 4,300 tokens per frame.
- 3DILG keeps a compact irregular set of latents chosen by farthest-point sampling near surfaces.
- Voxel Mamba keeps only non-empty voxels and serializes them along a Hilbert curve for a linear-time state-space model.
All three choose tokens by a fixed rule and make no claim about what the tokens represent. Here the selection is learned from the task, and it is backed by a recovery guarantee.
Window-based designs are the main compute competitors that do not sparsify. D-Former alternates local windows with dilated global sampling. VeloxSeg pairs small salient windows with larger ones at near-linear cost. UNETR uses coarse $16^3$ patches. These solve the cost problem but leave the tokens unconstrained.
Three papers constrain what we can claim:
- On Identifiability in Transformers proves that attention weights are not identifiable once the token count exceeds the head dimension. With $K \approx 1{,}700$ that is still the case. We therefore make no claim about the attention maps. The identifiable objects are the gated motif activations that feed attention, which that result does not cover.
- The 3D causal counterfactual models (Latent Causal Modeling for 3D Brain MRI Counterfactuals, Causal Image Synthesis of Brain MR in 3D) intervene in VQ-VAE or StyleGAN latents that have no identifiability guarantee. They show the need for trustworthy volumetric latents. They target a different goal, generation with a causal graph, which we do not attempt here.
- Auxiliary-variable identifiability (iVAE) and view-invariant slots (VISA) reach identifiability through extra observed variables or multiple views. The sparsity route needs neither, which matters for single-scan clinical data.
Research Program
Architecture. The model is a shallow 3D convolutional motif encoder with a receptive field of a few voxels, producing $M$ motif channels per location. Sparling's quantile gate keeps the top $\delta$ fraction of activations across the whole volume, with $\delta$ annealed from about $10^{-1}$ down to a target of $10^{-3}$ to $10^{-4}$. Each surviving activation becomes a token made of its motif vector plus a 3D positional encoding of its coordinates. A standard transformer processes these tokens. The head depends on the task:
- For volume-level labels, a pooled classifier.
- For segmentation, a light decoder that places token outputs back at their coordinates and upsamples them through skip connections from the local encoder.
In the segmentation variant, the skip path carries dense local features. That creates a leak around the bottleneck. We will test two versions. One restricts skips to the gated map. The other allows full skips, so we can measure whether the leak destroys identifiability.
Theory. Sparling's argument relies on locality and sparsity of the motif layer, and it limits what the downstream network may be. We will write out the 3D statement and check that a permutation-equivariant attention stack with positional encodings satisfies the downstream conditions. We will also bound how recovery degrades with the rate at which motifs overlap within one receptive field. That overlap rate is the parameter the original proof sets to zero.
Analysis. Across a grid of $\delta$ values and motif overlap rates, we measure three things: motif recovery, task accuracy, and memory. The expected result is a phase transition. Below some critical $\delta$, tied to the true motif density, recovery should jump, while compute keeps falling as $\delta^2$.
Baselines
The naive reference is a dense 3D convolutional encoder with the same local receptive field and no gate. We probe its channels for motif recovery by assigning channels to motif types with a best-case permutation. We also run a UNETR-style coarse-patch transformer as a second naive point. Both show what recovery looks like when nothing enforces it.
The strongest compute competitor on real data is VeloxSeg (Johnson-Lindenstrauss Lemma Guided Network for Efficient 3D Medical Segmentation). It reports 62.51% Dice on AutoPET-II, along with 1.66M parameters and 1.79 GFLOPs. AutoPET-II lesions are small and sparse, which is exactly the regime this method targets, so this is the most informative comparison. We will train VeloxSeg and our model on the same splits, with the same number of epochs, the same augmentation, and the same hyperparameter-search budget, and report peak GPU memory alongside accuracy. Beating it means matching its Dice within noise at equal or lower memory, while also delivering motif localization that VeloxSeg cannot be scored on. Winning on Dice alone is not the goal.
Voxel Mamba's Hilbert-curve state-space backbone is the strongest competitor that avoids attention cost entirely. We include it as the downstream module in a swap test. Its detection result on Waymo (79.6 L1 mAPH) is from a different task and is not a comparison target.
The ablations that isolate the key ingredient all hold the token count $K$ fixed and vary only how tokens are chosen:
- variance-map selection in the style of Omni-fMRI;
- farthest-point sampling in the style of 3DILG;
- random selection;
- a learned gate held at moderate sparsity (90%);
- an $L_1$ activation penalty in place of the hard quantile gate.
Sparling reports that $L_1$ and moderate sparsity fail on its 2D tasks. Our model counts as beating these ablations if it recovers motifs at above 90% localization and type accuracy on the synthetic benchmark, which is Sparling's 2D level, while every fixed-budget ablation stays well below it.
Why not just…?
Why not just prune tokens by variance or occupancy and run attention, as Omni-fMRI and Voxel Mamba already do? This is the real threat, because it reaches the same compute savings. Variance and occupancy rules favor texture and edges, not task-relevant structure, and nothing in Omni-fMRI or Voxel Mamba claims their tokens match generating factors. The fixed-$K$ ablation settles the question directly. If variance selection recovers motifs as well as the annealed gate, the identifiability contribution is empty and we will say so.
Why not just use windowed or dilated attention (D-Former, VeloxSeg) and probe the features afterward? These solve cost without sparsity. D-Former reports 88.83% Dice on Synapse. But dense, unconstrained features are exactly the setting where identifiability theory (iVAE) predicts non-unique solutions. Our naive-probe baseline measures how much best-case channel matching recovers from such features. If probing a windowed model matches our recovery, the gate is unnecessary.
Why not just add an $L_1$ penalty to a 3D CNN? Sparling reports that $L_1$ and KL penalties do not reach the 99.9% sparsity its theorem needs, and that at around 90% sparsity networks learn dense "cheating" codes. We re-test this in 3D inside the ablation, not by assumption. An $L_1$ model also does not produce a hard, small token set, so it gives no reduction in attention cost.
Why not just supervise the motifs with concept labels? Where labels exist, supervision is the right tool. But concept labels for every structure in every scan rarely exist, and the point of the theorem is recovery from end-task labels alone. The LIDC-IDRI experiment below uses nodule annotations only for evaluation, never for training.
Proposed Experiments
Synthetic 3D motif benchmark (the decisive test). We extend DigitCircle to $128^3$ and $256^3$ volumes. A small number of 3D glyphs, drawn from a fixed alphabet, are placed at random positions in noisy volumes. The volume label depends on a long-range relation between glyphs, for example their arrangement in space, so a purely local model cannot solve the task. Ground-truth glyph coordinates and types allow scoring motif localization and type accuracy under the best permutation. We vary three things: glyph overlap rate, $\delta$, and resolution.
This benchmark answers three questions:
- Does recovery exceed 90% at extreme sparsity while every fixed-$K$ ablation falls short?
- Does recovery hold when the attention downstream is replaced by a Voxel Mamba sequence model? It should, if identifiability comes from the gate and not from attention.
- How does recovery degrade as overlap grows?
We also report memory and throughput against dense attention at resolutions where dense attention still fits.
AutoPET-II lesion segmentation. We compare against VeloxSeg's reported 62.51% Dice under the fair-comparison protocol above. We add a lesion-level check that asks whether gated tokens land on annotated lesions, using a restricted-skip decoder so the bottleneck cannot be bypassed.
LIDC-IDRI weak localization. We train only on a scan-level "nodule present" label and measure how often gated tokens fall within annotated nodule locations. No reported result was found on Lacuna for weakly supervised nodule localization on LIDC-IDRI, so the baselines here are our own naive probe and the fixed-$K$ selectors.
Stress case. Synapse multi-organ segmentation involves large, dense organs and violates the sparsity assumption. It shows where the method should fail. We expect our model to trail D-Former's reported 88.83% average Dice, and measuring that gap marks the method's scope honestly.
References
- Sparling: End-to-End Spatial Concept Learning via Extremely Sparse Activations
- Omni-fMRI: A Universal Atlas-Free fMRI Foundation Model
- 3DILG: Irregular Latent Grids for 3D Generative Modeling
- Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection
- D-former: a U-shaped Dilated Transformer for 3D medical image segmentation
- Johnson-Lindenstrauss Lemma Guided Network for Efficient 3D Medical Segmentation
- UNETR: Transformers for 3D Medical Image Segmentation
- On Identifiability in Transformers
- Latent Causal Modeling for 3D Brain MRI Counterfactuals
- Causal Image Synthesis of Brain MR in 3D
- Variational Autoencoders and Nonlinear ICA: A Unifying Framework
- Identifiable Object Representations under Spatial Ambiguities