When we look at a three-dimensional object—whether it is a physical sculpture, a hand bone structure, or a human liver inside a CT scan—we intuitively understand it as a continuous, unified entity. However, early deep learning approaches to 3D data routinely broke this continuity. Because 2D convolutional neural networks (CNNs) were highly optimized and pre-trained on massive image datasets, the standard paradigm was to slice 3D volumes into a sequence of independent 2D images, process them individually, and aggregate the results.
As outlined by Dicente Cid in Overview of ImageCLEFtuberculosis 2019 - Automatic CT-based Report Generation and Tuberculosis Severity Assessment, this approach forces a stark trade-off between computational efficiency and contextual richness. Treating a 3D volume as a stack of disconnected 2D slices discards the spatial continuity along the depth axis. If an anatomical structure or a geometric anomaly spans multiple slices, a 2D network cannot natively perceive its volumetric boundaries, leading to fragmented features and inconsistent predictions.
To overcome this limitation, modern architectures extract features directly within the 3D domain. By preserving volumetric spatial context natively, these models capture the complex spatial hierarchies and inter-slice dependencies inherent in 3D data.
Moving from Pixels to Voxels: 3D Convolutions
The most direct way to preserve 3D context is to swap two-dimensional operations for their three-dimensional counterparts. Instead of sliding a flat $k \times k$ kernel across a 2D pixel grid, a 3D convolution slides a $k \times k \times k$ volumetric kernel through a grid of voxels (volume elements).
2D Convolution: [Kernel: K x K] 3D Convolution: [Kernel: K x K x K]
┌───┐ ┌───┐
│ │ ──> Slides in 2D (X, Y) │ │ ──> Slides in 3D (X, Y, Z)
└───┘ └───┘
(Processes individual slices) (Processes continuous volumes)
This structural shift allows the network to compute gradients and extract features across all three spatial axes simultaneously. As Qi Dou 0001 et al. demonstrate in 3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes, 3D kernels are uniquely capable of capturing inter-slice spatial dependencies. This capability is crucial when delineating structures with low contrast or highly irregular boundaries, where the context from adjacent slices resolves ambiguity that a single slice cannot.
By building on this volumetric foundation, architectures like the 3D U-Net, introduced by Özgün Çiçek et al. in 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation, extend the symmetrical analysis-synthesis structure into three dimensions. By replacing all operations—including max-pooling and up-convolutions—with 3D equivalents, the network leverages spatial continuity across the entire volume.
This native 3D awareness enables models to perform end-to-end learning directly on raw, unsegmented data. Sergey Korolev et al. in Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification and Soheil Esmaeilzadeh et al. in End-To-End Alzheimer's Disease Diagnosis and Biomarker Identification show that processing full volumes end-to-end allows networks to automatically discover diagnostic features and global context. This eliminates the need for destructive manual preprocessing steps like skull-stripping or rigid region-of-interest selection.
Adapting and Scaling to Transformers
While 3D CNNs excel at local feature extraction, scaling them to capture long-range global context introduces significant computational overhead. This has led to the adaptation of Vision Transformers (ViTs) for volumetric data. However, a naive application of transformers requires flattening an input into a 1D sequence of tokens, an operation that can severely disrupt physical coordinate alignment when applied to a 3D volume.
To combat this, recent architectures focus on maintaining dimensional integrity. Qikui Zhu et al. show in Class-consistent Contrastive Learning Driven Cross-dimensional Transformer for 3D Medical Image Classification that explicit dimensional correlation modeling is far more effective than flattening volumes into 1D sequences. By preserving these structural relationships, transformers can better capture the complex anatomical features required to identify subtle anomalies.
Similarly, in SegVol: Universal and Interactive Volumetric Medical Image Segmentation, Fan Bai et al. present a 3D Vision Transformer that treats volumetric data as a unified 3D entity rather than a sequence of independent slices. This design preserves critical spatial relationships across the depth dimension, allowing the model to maintain global anatomical context across the entire scan.
Naive Flattening: [ 3D Volume ] ──> [ Broken 1D Sequence ] (Loss of physical coordinate context)
Dimensional Integrity: [ 3D Volume ] ──> [ 3D-Aware Transformer ] (Preserves X, Y, Z spatial relationships)
Where full 3D transformers are computationally prohibitive, hybrid approaches bridge the gap:
- Causal Sequencing: Ibrahim Ethem Hamamci et al. demonstrate in CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging that modeling 3D medical volumes as sequential data using causal transformers preserves anatomical continuity across slices, treating the volumetric stack like a continuous temporal sequence.
- Lightweight Adapters: Cheng Chen 0013 et al. show in MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentation that injecting lightweight 3D adapters into pre-trained 2D transformer backbones allows models to capture inter-slice or temporal dependencies efficiently, preventing inconsistent segmentations without requiring a full 3D architecture from scratch.
Why Native 3D Representation Works
The fundamental advantage of native 3D feature extraction lies in its alignment with the physical reality of the data. This spatial fidelity manifests across several distinct domains:
1. Anatomical and Geometric Pre-training
Transfer learning from 2D natural image datasets (like ImageNet) provides broad feature diversity but lacks an understanding of 3D depth and geometry. In Learning Semantics-enriched Representation via Self-discovery, Self-classification, and Self-restoration, Fatemeh Haghighi et al. demonstrate that pre-training models directly on 3D volumetric data provides a substantial performance advantage over 2D transfer learning. Capturing 3D spatial relationships natively is more critical for volumetric tasks than the broad feature diversity found in general-purpose 2D datasets. This insight is central to building generalist foundation models for radiology, where Chaoyi Wu et al. show in Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical data that native 3D spatial awareness significantly outperforms 2D-based architectures in interpreting complex medical volumes.
2. High-Fidelity Interpretability and Saliency
When explaining model decisions, mapping 2D saliency methods onto 3D data often yields dispersed, noisy, and imprecise heatmaps. The authors of SE3D: A Framework For Saliency Method Evaluation In 3D Imaging show that 3D-native saliency architectures process volumetric data as a continuous spatial entity, capturing true three-dimensional connectivity and spatial dependencies. This prevents the artifacting common in naive 2D-to-3D extensions.
3. Non-Image Volumetric Fields
The necessity of 3D spatial context extends beyond traditional imaging to abstract volumetric grids. For example, in radiation oncology, Bin Liang et al. introduce "dosiomics" in Dosiomics: Extracting 3D Spatial Features From Dose Distribution to Predict Incidence of Radiation Pneumonitis. By extracting high-dimensional spatial features directly from 3D radiation dose grids, their approach captures the local heterogeneity and spatial relationships of energy distribution between voxels, significantly outperforming traditional one-dimensional dose summaries. Similarly, in computer vision, Zhirong Wu et al. demonstrate in 3D ShapeNets: A deep representation for volumetric shapes that applying 3D convolutional filters to volumetric shape data enables a generative framework to learn a hierarchical progression of geometric primitives—ranging from simple surfaces to complex object parts.
By treating the third dimension as an intrinsic property of the architecture rather than an afterthought, native 3D volumetric feature extraction ensures that deep learning models preserve the structural, anatomical, and geometric integrity of the world they are trying to understand.