Standard self-attention is a victim of its own success. While it excels at capturing global dependencies in 2D images, its computational cost scales quadratically ($O(N^2)$) with the number of input tokens. When moving from 2D images to 3D volumes—such as CT scans, MRIs, or hyperspectral cubes—the number of "voxels" explodes. A modest $128 \times 128 \times 128$ volume contains over two million tokens, making standard 3D self-attention mathematically impossible for even the most powerful GPUs.
To bridge this gap, researchers have converged on a shared strategy: Factorized Axial Attention. Instead of comparing every voxel to every other voxel in a massive 3D space, these models decompose volumetric relationships into independent, orthogonal passes along the height, width, and depth axes.
The Quadratic Wall in 3D
In a 2D Transformer, attention allows a pixel at the top-left to "talk" to a pixel at the bottom-right. In 3D medical imaging, this global context is even more critical; a lesion in one slice of an MRI might only be identifiable by looking at the anatomical structure several slices away.
However, as noted by Loic Themyr in the FINE transformer, the memory overhead of full 3D attention crashes standard hardware. The research direction across these 30 papers is the pursuit of "3D awareness" without the "3D cost."
Decomposing the Volume
The core intuition behind this research is that 3D spatial context can be effectively approximated by looking at the data from three orthogonal perspectives.
- Axial Fusion: Xiangyi Yan and colleagues introduced the AFTer-UNet, which factorizes 3D attention into intra-slice (spatial) and inter-slice (axial depth) operations. By processing these dimensions separately, the complexity drops from quadratic to linear relative to the volume size.
- Triple-View and Plane-Based Attention: Rather than treating the volume as a single block, models like VD-Former by Haofeng Li and EfficientMorph by Abu Zahid Bin Aziz decompose correlations into sequential 2D views (sagittal, coronal, and axial).
- Dimensional Decomposition: Qikui Zhu demonstrates in the Cross-dimensional Transformer that modeling height, width, and depth correlations independently reduces memory requirements by orders of magnitude while maintaining the ability to capture high-resolution volumetric details.
Reconstructing Global Context
If we only look at one axis at a time, do we lose the "big picture"? The evidence suggests the opposite. By stacking these factorized layers, the receptive field expands rapidly.
In CSwin UNet, Jacob F. Wynne uses "cross-shaped" window attention—horizontal, vertical, and longitudinal stripes—to ensure that every voxel eventually "sees" every other voxel through a series of orthogonal projections. This is analogous to how a CT scanner reconstructs a 3D object from 2D projections; the model reconstructs 3D dependencies from 1D or 2D attention passes.
This mechanism isn't limited to medical voxels. Lunhao Duan applied a similar "disassembled" approach to 3D point clouds in ConDaFormer, proving that decomposing cubic windows into orthogonal planes is a universal solution for high-dimensional spatial understanding.
Why It Works: Efficiency and Precision
The success of this direction stems from two main factors:
- Memory Efficiency: Jeremy Kim showed that RSANet uses 28x less memory than traditional non-local blocks. This allows researchers to train on consumer-grade GPUs rather than industrial clusters.
- Feature Focus: By separating dimensions, the model can learn distinct features for different axes. For example, in SAX-NeRF, Jiahao Wang uses a "Lineformer" to treat X-rays as structural line segments, capturing the inherent geometry of the data more precisely than a generic 3D cube would.
By breaking the 3D volume into its constituent axes, factorized axial attention transforms the "curse of dimensionality" into a structured, efficient, and highly accurate framework for volumetric deep learning.