In the transition from 2D to 3D deep learning, researchers have encountered a fundamental "locality bottleneck." Traditional 3D Convolutional Neural Networks (CNNs) operate like a small flashlight in a dark room: they can see textures and edges in a tiny neighborhood, but they struggle to understand how a structure at the top of a scan relates to one at the bottom.
A new paradigm is emerging that replaces these local filters with Volumetric Transformers. By treating 3D volumes as sequences of information, these models use self-attention to capture "long-range dependencies"—the global context necessary to understand complex anatomical and spatial relationships.
The Locality Bottleneck in 3D
In 3D medical imaging (CT, MRI) or LiDAR point clouds, the most critical information often lies in the relationship between distant parts. For example, segmenting a single organ requires knowing where other organs are located relative to it.
Standard 3D CNNs struggle here because their receptive fields grow slowly. To "see" the whole volume, you need many layers, which leads to vanishing gradients and massive computational costs. As noted in Deep Learning based 3D Segmentation: A Survey by Yong He and Ajmal Mian, the field is shifting toward Transformers because they can model these dependencies across the entire volume simultaneously.
Reformulating Volumes as Sequences
The core intuition behind this research direction is to treat a 3D volume not as a rigid grid, but as a sequence of "tokens."
In the UNETR architecture, Ali Hatamizadeh and Dong Yang reformulate 3D segmentation as a sequence-to-sequence task. They divide a 3D volume into a sequence of small 3D patches. Because every patch in the sequence can "attend" to every other patch via self-attention, the model learns a global representation from the very first layer.
This sequence-based approach allows for creative cross-domain learning. For instance, Alexander Ke and Pranav Rajpurkar demonstrated in Video pretraining advances 3D deep learning on chest CT tasks that we can treat the depth dimension of a CT scan like the time dimension in a video. By pre-training on natural videos, a model learns how objects move and change through a sequence, which translates directly to understanding how anatomy changes across CT slices.
Efficiency through Shifted Windows
The primary challenge of applying Transformers to 3D data is the "quadratic explosion": if you have twice as many voxels, the attention calculation becomes four times as expensive. To solve this, many researchers have adopted Shifted Window mechanisms.
In CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection, Jie Liu and Yixiao Zhang utilize a hierarchical 3D transformer that restricts attention to local windows but "shifts" those windows in subsequent layers. This allows information to propagate globally across the volume without the impossible memory requirements of dense global attention. This same efficiency allows Elham Abouei to use 3D transformers within diffusion models to generate synthetic CT scans from MRI, maintaining anatomical consistency across the entire generated volume.
Hybrid Architectures and Refinement
Pure Transformers are excellent at global context but can sometimes lose the "fine-grained" details that CNNs excel at. This has led to the rise of hybrid models like 3D TransUNet.
Jieneng Chen and Yongyi Lu found that the best results come from using a Transformer encoder to understand the "big picture" (the relationships between organs) and a convolutional decoder to refine the sharp boundaries of those organs. Similarly, Rui Nian and Guoyao Zhang developed the 3D Brainformer, which uses a "Fusion-Head" attention mechanism to bridge the gap between these global and local features.
Learning from Sparsity and Completion
Beyond medical imaging, these concepts are being applied to sparse 3D data like LiDAR. In SCP: Scene Completion Pre-training for 3D Object Detection, Yan Xia and Daniel Cremers show that we can teach a model about 3D space by asking it to "fill in the blanks" of a sparse scene.
By using voxel-based transformers to predict missing parts of a 3D environment, the model develops a deep understanding of spatial layout. This self-supervised pre-training allows detectors to achieve high performance with only 20% of the usual labeled data, proving that the "global understanding" provided by transformers is a powerful substitute for massive human annotation.